Emotion Recognition Through Facial Expressions: A Machine Learning Perspective in Mobile Multimedia
DOI:
https://doi.org/10.13052/jmm1550-4646.2114Keywords:
Emotion detection, human-computer interaction, advanced machine learning, facial expressions, feature extractionAbstract
Facial expression-based emotion detection is very attractive because of the possibilities in security systems, mental health monitoring, and human-computer interaction. Even with the progress in accuracy in real-world settings, issues such as the lack of balanced datasets and the inability to differentiate between faint or superimposed emotions continue to plague it. This study aims to bridge these constraints by developing a CNN-based model that would be able to recognize face emotions reliably and be utilized in real-time situations, such as webcam integration. The Affect Net dataset, which is a comprehensive collection of over a million facial photos labeled with the seven major emotions of anger, disgust, fear, happiness, neutrality, sadness, and surprise, was used to train the proposed model. Other pre-processing data techniques used include grayscale conversion, normalization, scaling, and data supplementation to increase the robustness of the model. Using metrics like accuracy and loss trends for evaluation, the model demonstrated efficiency stability at around the 30th training phase. When the model is compared to existing models, this proposed model can attain the competitive level of accuracy up to approximately 60%. It also has the potential to run in real applications through its webcam integration. While the model can differentiate between various clear-cut emotions, it becomes ineffective at identifying subtle emotions, which include “Fear” and “Neutral” majorly because of unbalanced data and the subtleness of these expressions.
Downloads
References
Balakrishnan, J., and Dwivedi, Y. K. (2021). Role of cognitive absorption in building user trust and experience. Psychology & Marketing, 38(4), 643–668.
Mavropoulos, T., Symeonidis, S., Tsanousa, A., Giannakeris, P., Rousi, M., Kamateri, E., … and Kompatsiaris, I. “Smart integration of sensors, computer vision and knowledge representation for intelligent monitoring and verbal human-computer interaction.” Journal of Intelligent Information Systems, vol. 57, no. 2, pp. 321–345, 2020.
Hollender, N., Hofmann, C., Deneke, M., and Schmitz, B. “Integrating cognitive load theory and concepts of human–computer interaction.” Computers in human behavior, vol. 26, no. 6, pp. 1278–1288, 2020.
Glodek, M., Honold, F., Geier, T., Krell, G., Nothdurft, F., Reuter, S., … and Schwenker, F. “Fusion paradigms in cognitive technical systems for human–computer interaction”. Neurocomputing, vol. 161, pp. 17–37, 2015.
Ali, S. I., Jain, S., Lal, B., and Sharma, N. “A framework for modeling and designing of intelligent and adaptive interfaces for human computer interaction”. International Journal of Applied Information Systems (IJAIS), 2012.
Dutta, S., Mishra, B. K., Mitra, A., and Chakraborty, A. (2022). An analysis of emotion recognition based on GSR signal. ECS Transactions, 107(1), 12535.
Val-Calvo, M., Álvarez-Sánchez, J. R., Ferrández-Vicente, J. M., Díaz-Morcillo, A., and Fernández-Jover, E. “Real-time multi-modal estimation of dynamically evoked emotions using EEG, heart rate and galvanic skin response.” International Journal of Neural Systems, vol. 30, no. 4, pp. 2050013, 2020.
Raheel, A., Majid, M., Alnowami, M., and Anwar, S. M. “Physiological sensors-based emotion recognition while experiencing tactile enhanced multimedia”. Sensors, vol. 20, no. 14, pp. 4037, 2020.
Samadiani, N., Huang, G., Cai, B., Luo, W., Chi, C. H., Xiang, Y., and He, J. “A review on automatic facial expression recognition systems assisted by multimodal sensor data”. Sensors, vol. 19, no. 8, pp. 1863, 2019.
Witkower, Z., and Tracy, J. L. “Bodily communication of emotion: Evidence for extrafacial behavioral expressions and available coding systems”. Emotion Review, vol. 11, no. 2, pp. 184–193, 2019.
Ahmed, F., Bari, A. H., and Gavrilova, M. L. “Emotion recognition from body movement”. IEEE Access, vol. 8, pp. 11761–11781, 2019.
Sisk, C. A., Remington, R. W., and Jiang, Y. V. “Mechanisms of contextual cueing: A tutorial review.” Attention, Perception, and Psychophysics, vol. 81, pp. 2571–2589, 2019.
Cummings, M., and Stimpson, A. “Identifying critical contextual design cues through a machine learning approach”. AI Magazine, vol. 40, no. 4, pp. 28–39, 2019.
Martins, R., Bersan, D., Campos, M. F., and Nascimento, E. R. “Extending maps with semantic and contextual object information for robot navigation: a learning-based framework using visual and depth cues.” Journal of Intelligent & Robotic Systems, vol. 99, no. 3, pp. 555–569, 2020.
Ortega, A., Frossard, P., Kovačević, J., Moura, J. M., and Vandergheynst, P. “Graph signal processing: Overview, challenges, and applications.” Proceedings of the IEEE, vol. 106, no. 5, pp. 808–828, 2018.
Zhang, J. A., Liu, F., Masouros, C., Heath, R. W., Feng, Z., Zheng, L., and Petropulu, A. “An overview of signal processing techniques for joint communication and radar sensing.” IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 6, pp. 1295–1315, 2021.
Makowski, D., Pham, T., Lau, Z. J., Brammer, J. C., Lespinasse, F., Pham, H., and Chen, S. A. “NeuroKit2: A Python toolbox for neurophysiological signal processing”. Behavior research methods, pp. 1–8, 2021.
Zebari, R., Abdulazeez, A., Zeebaree, D., Zebari, D., and Saeed, J. “A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction.” Journal of Applied Science and Technology Trends, vol. 1, no. 1, pp. 56–70, 2020.
Alelyani, S., Tang, J., and Liu, H. “Feature selection for clustering: A review”. Data Clustering, pp. 29–60, 2018.
Chen, Z., Zhao, P., Li, F., Leier, A., Marquez-Lago, T. T., Wang, Y., … and Song, J. “Feature: a python package and web server for features extraction and selection from protein and peptide sequences”. Bioinformatics, vol. 34, no. 14, pp. 2499–2502, 2018.
Zhang, J., Yin, Z., Chen, P., and Nichele, S. “Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review.” Information Fusion, vol. 59, pp. 103–126, 2020.
Sarkar, P., and Etemad, A. “Self-supervised ECG representation learning for emotion recognition.” IEEE Transactions on Affective Computing, vol. 13, no. 3, pp. 1541–1554, 2020.
Houssein, E. H., Hammad, A., and Ali, A. A. “Human emotion recognition from EEG-based brain–computer interface using machine learning: a comprehensive review.” Neural Computing and Applications, vol. 34, no. 15, pp. 12527–12557, 2022.
Bau, D., Zhu, J. Y., Strobelt, H., Lapedriza, A., Zhou, B., and Torralba, A. “Understanding the role of individual units in a deep neural network.” Proceedings of the National Academy of Sciences, vol. 117, no. 48, pp. 30071–30078, 2020.
Chen, M., Challita, U., Saad, W., Yin, C., and Debbah, M. “Artificial neural networks-based machine learning for wireless networks: A tutorial”. IEEE Communications Surveys & Tutorials, vol. 21, no. 4, pp. 3039–3071, 2019.
Voigtlaender, F. (2023). The universal approximation theorem for complex-valued neural networks. Applied and computational harmonic analysis, 64, 33–61.
Zhang, Shiqing, Yijiao Yang, Chen Chen, Xingnan Zhang, Qingming Leng, and Xiaoming Zhao. “Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects.” Expert Systems with Applications 237, 121692, 2024.
Lian, Z., Sun, L., Sun, H., Chen, K., Wen, Z., Gu, H., Liu, B. and Tao, J.,. “Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition”. Information Fusion, 108, p. 102367, 2024.
Wei, Jie, Guanyu Hu, Xinyu Yang, Anh Tuan Luu, and Yizhuo Dong. “Learning facial expression and body gesture visual information for video emotion recognition.” Expert Systems with Applications, 237, 121419, 2024.
Xu, FeiFan, Deng Pan, Haohao Zheng, Yu Ouyang, Zhe Jia, and Hong Zeng. “EESCN: A novel spiking neural network method for EEG-based emotion recognition.” Computer methods and programs in biomedicine 243, 107927, 2024.
