An Approach for Learner Categorization Based on Emotions in Intelligent Adaptive E-Learning Environment


  • Myneni Madhu Bala Institute of Aeronautical Engineering, Hyderabad, India
  • Haritha Akkineni P.V.P Siddhartha Institute of Technology, Vijayawada, India
  • Chennupalli Srinivasulu Institute of Aeronautical Engineering, Hyderabad, India



Convolution Neural Network, E learning system, Face emotion recognition


The pandemic across the globe has constrained the change from a conventional face to face to e-learning platforms. The most challenging task during online learning is to be aware and support the emotional side of students. In existing environments, the emotion of the listener consideration is lagging. This can be provided by capturing the emotions of the listener through facial expressions. In general, the most common facial expressions are happy, sad, anger, fear, disgust, neutral and surprise. This knowledge can be used to classify different listeners. Hence in this article, we proposed a novel approach to identify an emotion based learner category in the development of Intelligent Adaptive E-Learning Environment by using Convolution Neural Network. The major work is composed of emotion detection model and learner categorization. The emotion detection model is trained by using a standard FER2013 dataset and it is extended with live streams of learners. The results of emotion detection model are extended to categorize the learners by fusing emotions and comprehend as Active, Evaluative, Passive and Non-Listener. The proposed model is trained using 100 epochs and achieved an accuracy of 94.44% in the training phase. This knowledge helps to interpret learner’s participation in e-learning environment.


Download data is not yet available.

Author Biographies

Myneni Madhu Bala, Institute of Aeronautical Engineering, Hyderabad, India

Myneni Madhu Bala is currently working as a professor and Dean of computational studies at Institute of Aeronautical Engineering, Hyderabad. She received her Ph.D in Computer Science and Engineering from JNTUH. She has Twenty one years of academic and research experience Her research interests are Data Science frameworks, Image Mining, Text mining, Machine learning, Artificial Intelligence, Deep Learning and Data Analytics. She has published about 57 articles in reputed Journals indexed in SCOPUS, SCI etc. She has published 2 patents. She has received grants from AICTE for organizing Short Term Training Programs and Faculty Development Programs. She is a reviewer for Elsevier, Springer and more indexed journals. She acted as sessionchair, organising member, and advisory member for various International Conferences. She delivered various invited talks on Data Modelling, Data Science and Analytics. She is a Life member of professional bodies like CSI and ISTE, Sr. Member for IEEE, WIE & International association IAENG, ICST, SDIWC.

Haritha Akkineni, P.V.P Siddhartha Institute of Technology, Vijayawada, India

Haritha Akkineni is currently an associate professor in Information Technology at PVP Siddhartha Institute of Technology, Vijayawada. She received her Ph.D in Computer Science and Engineering. She is working in the area of Opinion Mining and Data Sciences. She has twelve years of academic and research experience Her research interests are Data Science, Image Mining, Artificial Intelligence, Data Analytics, Deep Learning and Machine Learning. She has published about 35 papers in reputed Journals like SCOPUS UGC etc. She has published 2 patents. She has received grants from AICTE for organizing Short Term Training Programs. She is a reviewer for SCOPUS indexed journals. She authored a book on Opinion Mining. She acted as Workshop/tutorial chair for various International Conferences. She delivered various invited talks.

Chennupalli Srinivasulu, Institute of Aeronautical Engineering, Hyderabad, India

Chennupalli Srinivasulu is working as Professor of CSE at Institute of Aeronautical Engineering (IARE). He has 24 Years of experience in Teaching and Industry. His Area of Interest is Parallel Computing, Soft Computing, Software Engineering etc. He has published more than 38 Publications in reputed International Journals.


C. Mega, L. Ronconi, and R. De Beni, What makes a good student? How emotions, self-regulated learning, and motivation contribute to academic achievement., J. Educ. Psychol., vol. 106, no. 1, p. 121, 2014.

C. H. Wu, New technology for developing facial expression recognition in e-learning, in 2016 Portland International Conference on Management of Engineering and Technology (PICMET), pp. 1719–1722, 2016.

A. Dhall, R. Goecke, J. Joshi, K. Sikka, and T. Gedeon, Emotion recognition in the wild challenge 2014: baseline, data and protocol, in Proceedings of the 16th International Conference on Multimodal Interaction, pp. 461–466, ACM, Istanbul Turkey, November 2014.

J. Li, K. Jin, D. Zhou, N. Kubota, and Z. Ju, Attention mechanism-based CNN for facial expression recognition, Neurocomputing, vol. 411, pp. 340–350, 2020.

J. Shao and Y. Qian, Three convolutional neural network models for facial expression recognition in the wild, Neurocomputing, vol. 355, pp. 82–92, 2019.

F. Benmarrakchi, J. E. Kafi, A. Elhore, and S. Haie, Exploring the use of the ICT in supporting dyslexic students preferred learning styles : A preliminary evaluation, Educ. Inf. Technol., pp. 1–19, Oct. 2016.

O. El Hammoumi, F. Benmarrakchi, N. Ouherrou, J. El Kafi and A. El Hore, Emotion Recognition in E-learning Systems, 2018 6th International Conference on Multimedia Computing and Systems (ICMCS), Rabat, Morocco, pp. 1–6, doi: 10.1109/ICMCS.2018.8525872, 2018.

Liu Y, Li Y, Ma X, Song R, Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17:712, 2017.

Xing Y, Luo W, Facial expression recognition using local Gabor features and adaboost classifiers. In: 2016 international conference on progress in informatics and computing (pic). IEEE, pp. 228–232, 2016.

Gupta O, Raviv D, Raskar R, Multi velocity neural networks for facial expression recognition in videos. IEEE Trans Affect Comput 10: 290–296, 2017.

Lopes AT, De Aguiar E, Oliveira-Santos T, A facial expression recognition system using convolutional networks. In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images. IEEE, pp. 273–280, 2015.

Mehendale, N. Facial emotion recognition using convolutional neural networks (FERC). SN Appl. Sci. 2, 446,, 2020.

I. J. Goodfellow, D. Erhan, P. L. Carrier et al., Challenges in representation learning: a report on three machine learning contests, Neural Information Processing, Springer, Berlin, Germany, pp. 117–124, 2013.

Shan, K., Guo, J., You, W., Lu, D., and Bie, R, Automatic facial expression recognition based on a deep convolutional-neural-network structure. 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA), 123–128, 2017.

Mohammed Megahed, Ammar Mohammed, Modeling adaptive e-learning environment using facial expressions and fuzzy logic, Expert Systems with Applications, Volume 157, 113460, ISSN 0957- 174, 2020.

J. M. Harley, S. P. Lajoie, C. Frasson, and N. C. Hall, An Integrated Emotion-Aware Framework for Intelligent Tutoring Systems, in Artificial Intelligence in Education, pp. 616–619, 2015.

Khalfallah, J., Slama, J. B. H., Facial expression recognition for intelligent tutoring systems in remote laboratories platform. Procedia computer science, 73, 274–281. doi: 10.1016/j.procs.2015.12.030, 2015.

Zatarain-Cabada, R., Barrón-Estrada, M. L., García-Lizárraga, J., Muñoz Sandoval, G., Ríos-Félix, J. M. Java tutoring system with facial and text emotion recognition. Research in Computing Science, 106, 49–58, 2015.

Phan-Xuan, H., Le-Tien, T., Nguyen-Tan, S., Fpga platform applied for facial expression recognition system using convolutional neural networks. Procedia computer science, 151, 651–658. doi: 10.1016/j.procs.2019.04.087, 2019.

Ayvaz, U., Gürüler, H., Devrim, M. O., Use of facial emotion recognition in e-learning systems. Information Technologies and Learning Tools, 60(4), 95–104. doi: 10.33407/itlt.v60i4.1743, 2017.

H.-D. Nguyen, S.-H. Kim, G.-S. Lee, H.-J. Yang, I.-S. Na and S.-H. Kim, “Facial Expression Recognition Using a Temporal Ensemble of Multi-Level Convolutional Neural Networks,” in IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 226–237, 1 Jan.–March 2022, doi: 10.1109/TAFFC.2019.2946540.

C. Shi, C. Tan and L. Wang, “A Facial Expression Recognition Method Based on a Multibranch Cross Connection Convolutional Neural Network,” in IEEE Access, vol. 9, pp. 39255–39274, 2021, doi: 10.1109/ACCESS.2021.3063493.

C. Liu, K. Hirota, J. Ma, Z. Jia and Y. Dai, “Facial Expression Recognition Using Hybrid Features of Pixel and Geometry,” in IEEE Access, vol. 9, pp. 18876–18889, 2021, doi: 10.1109/ACCESS.2021.3054332.

Agrawal, A., Mittal, N., Using CNN for facial expression recognition: A study of the effects of kernel size and number of filters on accuracy. The Visual computer. doi: 10.1007/s00371-019-01630-9, 2019.

N.M. Deepika and Myneni Madhu Bala and Ravi Kumar, ”Design and implementation of intelligent virtual laboratory using RASA framework”, Materials Today: Proceedings 2021 ISSN 2214-7853., 2021.

Monika Dubey, Prof. Lokesh Singh, Automatic Emotion Recognition Using Facial Expression: A Review, International Research Journal of Engineering and Technology, Volume: 03 e-ISSN: 2395-0056, 2016.

C. Jain, K. Sawant, M. Rehman and R. Kumar, “Emotion Detection and Characterization using Facial Features,” 2018 3rd International Conference and Workshops on Recent Advances and Innovations in Engineering (ICRAIE), Jaipur, India, pp. 1–6, doi: 10.1109/ICRAIE.2018.8710406, 2018.

Krithika, L. B., Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia computer science, 85, 767–776. doi: 10.1016/j.procs.2016.05.264, 2016.

Dandamudi, Rohit and Myneni, Madhu Bala, A Smart Personalized Learning Environment Through Social QA System (December 4, 2020). e-journal – First Pan IIT International Management Conference 2018.

Breuer R, Kimmel R, A deep learning perspective on the origin of facial expressions. arXivPrepr arXiv170501842, 2017.

Fan Y, Lu X, Li D, Liu Y, Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction. pp. 445–450, 2016.

Mollahosseini A, Chan D, Mahoor MH, Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV). IEEE, pp. 1–10, 2016.