An Approach for Learner Categorization Based on Emotions in Intelligent Adaptive E-Learning Environment
DOI:
https://doi.org/10.13052/jmm1550-4646.18611Keywords:
Convolution Neural Network, E learning system, Face emotion recognitionAbstract
The pandemic across the globe has constrained the change from a conventional face to face to e-learning platforms. The most challenging task during online learning is to be aware and support the emotional side of students. In existing environments, the emotion of the listener consideration is lagging. This can be provided by capturing the emotions of the listener through facial expressions. In general, the most common facial expressions are happy, sad, anger, fear, disgust, neutral and surprise. This knowledge can be used to classify different listeners. Hence in this article, we proposed a novel approach to identify an emotion based learner category in the development of Intelligent Adaptive E-Learning Environment by using Convolution Neural Network. The major work is composed of emotion detection model and learner categorization. The emotion detection model is trained by using a standard FER2013 dataset and it is extended with live streams of learners. The results of emotion detection model are extended to categorize the learners by fusing emotions and comprehend as Active, Evaluative, Passive and Non-Listener. The proposed model is trained using 100 epochs and achieved an accuracy of 94.44% in the training phase. This knowledge helps to interpret learner’s participation in e-learning environment.
Downloads
References
C. Mega, L. Ronconi, and R. De Beni, What makes a good student? How emotions, self-regulated learning, and motivation contribute to academic achievement., J. Educ. Psychol., vol. 106, no. 1, p. 121, 2014.
C. H. Wu, New technology for developing facial expression recognition in e-learning, in 2016 Portland International Conference on Management of Engineering and Technology (PICMET), pp. 1719–1722, 2016.
A. Dhall, R. Goecke, J. Joshi, K. Sikka, and T. Gedeon, Emotion recognition in the wild challenge 2014: baseline, data and protocol, in Proceedings of the 16th International Conference on Multimodal Interaction, pp. 461–466, ACM, Istanbul Turkey, November 2014.
J. Li, K. Jin, D. Zhou, N. Kubota, and Z. Ju, Attention mechanism-based CNN for facial expression recognition, Neurocomputing, vol. 411, pp. 340–350, 2020.
J. Shao and Y. Qian, Three convolutional neural network models for facial expression recognition in the wild, Neurocomputing, vol. 355, pp. 82–92, 2019.
F. Benmarrakchi, J. E. Kafi, A. Elhore, and S. Haie, Exploring the use of the ICT in supporting dyslexic students preferred learning styles : A preliminary evaluation, Educ. Inf. Technol., pp. 1–19, Oct. 2016.
O. El Hammoumi, F. Benmarrakchi, N. Ouherrou, J. El Kafi and A. El Hore, Emotion Recognition in E-learning Systems, 2018 6th International Conference on Multimedia Computing and Systems (ICMCS), Rabat, Morocco, pp. 1–6, doi: 10.1109/ICMCS.2018.8525872, 2018.
Liu Y, Li Y, Ma X, Song R, Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17:712, 2017.
Xing Y, Luo W, Facial expression recognition using local Gabor features and adaboost classifiers. In: 2016 international conference on progress in informatics and computing (pic). IEEE, pp. 228–232, 2016.
Gupta O, Raviv D, Raskar R, Multi velocity neural networks for facial expression recognition in videos. IEEE Trans Affect Comput 10: 290–296, 2017.
Lopes AT, De Aguiar E, Oliveira-Santos T, A facial expression recognition system using convolutional networks. In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images. IEEE, pp. 273–280, 2015.
Mehendale, N. Facial emotion recognition using convolutional neural networks (FERC). SN Appl. Sci. 2, 446, https://doi.org/10.1007/s42452-020-2234-1, 2020.
I. J. Goodfellow, D. Erhan, P. L. Carrier et al., Challenges in representation learning: a report on three machine learning contests, Neural Information Processing, Springer, Berlin, Germany, pp. 117–124, 2013.
Shan, K., Guo, J., You, W., Lu, D., and Bie, R, Automatic facial expression recognition based on a deep convolutional-neural-network structure. 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA), 123–128, 2017.
Mohammed Megahed, Ammar Mohammed, Modeling adaptive e-learning environment using facial expressions and fuzzy logic, Expert Systems with Applications, Volume 157, 113460, ISSN 0957- 174, 2020.
J. M. Harley, S. P. Lajoie, C. Frasson, and N. C. Hall, An Integrated Emotion-Aware Framework for Intelligent Tutoring Systems, in Artificial Intelligence in Education, pp. 616–619, 2015.
Khalfallah, J., Slama, J. B. H., Facial expression recognition for intelligent tutoring systems in remote laboratories platform. Procedia computer science, 73, 274–281. doi: 10.1016/j.procs.2015.12.030, 2015.
Zatarain-Cabada, R., Barrón-Estrada, M. L., García-Lizárraga, J., Muñoz Sandoval, G., Ríos-Félix, J. M. Java tutoring system with facial and text emotion recognition. Research in Computing Science, 106, 49–58, 2015.
Phan-Xuan, H., Le-Tien, T., Nguyen-Tan, S., Fpga platform applied for facial expression recognition system using convolutional neural networks. Procedia computer science, 151, 651–658. doi: 10.1016/j.procs.2019.04.087, 2019.
Ayvaz, U., Gürüler, H., Devrim, M. O., Use of facial emotion recognition in e-learning systems. Information Technologies and Learning Tools, 60(4), 95–104. doi: 10.33407/itlt.v60i4.1743, 2017.
H.-D. Nguyen, S.-H. Kim, G.-S. Lee, H.-J. Yang, I.-S. Na and S.-H. Kim, “Facial Expression Recognition Using a Temporal Ensemble of Multi-Level Convolutional Neural Networks,” in IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 226–237, 1 Jan.–March 2022, doi: 10.1109/TAFFC.2019.2946540.
C. Shi, C. Tan and L. Wang, “A Facial Expression Recognition Method Based on a Multibranch Cross Connection Convolutional Neural Network,” in IEEE Access, vol. 9, pp. 39255–39274, 2021, doi: 10.1109/ACCESS.2021.3063493.
C. Liu, K. Hirota, J. Ma, Z. Jia and Y. Dai, “Facial Expression Recognition Using Hybrid Features of Pixel and Geometry,” in IEEE Access, vol. 9, pp. 18876–18889, 2021, doi: 10.1109/ACCESS.2021.3054332.
Agrawal, A., Mittal, N., Using CNN for facial expression recognition: A study of the effects of kernel size and number of filters on accuracy. The Visual computer. doi: 10.1007/s00371-019-01630-9, 2019.
N.M. Deepika and Myneni Madhu Bala and Ravi Kumar, ”Design and implementation of intelligent virtual laboratory using RASA framework”, Materials Today: Proceedings 2021 ISSN 2214-7853. doi.org/10.1016/j.matpr.2021.01.226, 2021.
Monika Dubey, Prof. Lokesh Singh, Automatic Emotion Recognition Using Facial Expression: A Review, International Research Journal of Engineering and Technology, Volume: 03 e-ISSN: 2395-0056, 2016.
C. Jain, K. Sawant, M. Rehman and R. Kumar, “Emotion Detection and Characterization using Facial Features,” 2018 3rd International Conference and Workshops on Recent Advances and Innovations in Engineering (ICRAIE), Jaipur, India, pp. 1–6, doi: 10.1109/ICRAIE.2018.8710406, 2018.
Krithika, L. B., Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia computer science, 85, 767–776. doi: 10.1016/j.procs.2016.05.264, 2016.
Dandamudi, Rohit and Myneni, Madhu Bala, A Smart Personalized Learning Environment Through Social QA System (December 4, 2020). e-journal – First Pan IIT International Management Conference 2018.
Breuer R, Kimmel R, A deep learning perspective on the origin of facial expressions. arXivPrepr arXiv170501842, 2017.
Fan Y, Lu X, Li D, Liu Y, Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction. pp. 445–450, 2016.
Mollahosseini A, Chan D, Mahoor MH, Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV). IEEE, pp. 1–10, 2016.
