EMOTION AND MOOD RECOGNITION IN RESPONSE TO VIDEO
Keywords:
Affective computing, mood, emotion recognition, video clip, EEGAbstract
This paper presents a subject-dependent homogenous emotion recognition method using electroencephalogram (EEG) signals in response to video contents, and a correlation between emotions and moods of subjects in resting state. In the recent years, there has been a trend towards recognizing emotions invoked from watching videos. Thus, in this study, two video clips with explicit emotional contents from movies and online resources were used, and the EEG results were recorded from four subjects as they watched these clips. The best accuracies of 60.71% for valence and 63.73% for arousal were obtained using a Mel-frequency cepstral coefficients (MFCC) and multilayer perceptron (MLP). The results show that MFCC and MLP techniques are applicable in emotion recognition. The result shows that the mood can be recognized from opened eyes or closed eyes experiment of a subject. Furthermore, the results demonstrated that a positive video content can stimulate a subject into being in positive emotional state even when the subject was in bad mood. The emotional state in response to watching a video was shown to be correlated with Self-Assessment Manikin analysis.
Downloads
References
R. Khosrowabadi, A. Wahab, K. K. Ang, and M. H. Baniasad, “Affective Computation on EEG
Correlates of Emotion from Musical and Vocal Stimuli,” Proceeding Int. Jt. Conf. Neural
Networks, Atlanta, Georg. USA, pp. 1590–1594, 2009.
J. J. M. Kierkels, M. Soleymani, and Thierry Pun, “Queries and Tags in Affect-Based Multimedia
Retrieval,” Int’l. Conf.Multimedia Expo, Spec. Sess. Implicit Tagging (ICME 2009), pp. 1436–
, 2009.
G. Lee, M. Kwon, S. Kavuri Sri, and M. Lee, “Emotion Recognition Based on 3D Fuzzy Visual
and EEG Features in Movie Clips,” Neurocomputing, vol. 144, pp. 560–568, Nov. 2014.
S. Koelstra and I. Patras, “Fusion of facial expressions and EEG for implicit affective tagging,”
Image Vis. Comput., vol. 31, no. 2, pp. 164–174, Feb. 2013.
J. J. M. Kierkels, B. B. A, R. De Drize, and C. H. Carouge, “A Bayesian Framework for Video
Affective Representation Mohammad Soleymani Guillaume Chanel Thierry Pun Computer Vision
and Multimedia Laboratory , Computer Science Department,” 2009.
A. Yazdani, J.-S. Lee, and T. Ebrahimi, “Implicit Emotional Tagging of Multimedia Using EEG
Signals and Brain Computer Interface,” Proc. first SIGMM Work. Soc. media - WSM ’09, p. 81,
M. Soleymani, M. Pantic, and T. Pun, “Multimodal Emotion Recognition in Response to Videos,”
IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 211–223, Apr. 2012.
M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A Multimodal Database for Affect
Recognition and Implicit Tagging,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 42–55, Jan.
J. Broekens and W.-P. Brinkman, “AffectButton: A method for Reliable and Valid Affective Selfreport,”
Int. J. Hum. Comput. Stud., vol. 71, no. 6, pp. 641–667, Jun. 2013.
G. Irie, T. Satou, A. Kojima, T. Yamasaki, and K. Aizawa, “Affective Audio-Visual Words and
Latent Topic Driving Model for Realizing Movie Affective Scene Classification,” IEEE Trans.
Multimed., vol. 12, no. 6, pp. 523–535, Oct. 2010.
D. Sanchez-Cortes, J.-I. Biel, S. Kumano, J. Yamato, K. Otsuka, and D. Gatica-Perez, “Inferring
Mood in Ubiquitous Conversational Video,” Proc. 12th Int. Conf. Mob. Ubiquitous Multimed. -
MUM ’13, pp. 1–9, 2013.
Y. Baveye, E. Dellandréa, C. Chamaret, and L. Chen, “LIRIS-ACCEDE: A Video Database for
Affective Content Analysis,” Affect. Comput. IEEE Trans., pp. 1–14, 2015.
K. Park, H. Choi, and K. Lee, “Emotion Recognition Based on The Asymmetric Left and Right
Activation,” Int. J. Med. Med. Sci., vol. 3, no. June, pp. 201–209, 2011.
M. Murugappan, R. Nagarajan, and Sazali Yaacob, “Combining Spatial Filtering and Wavelet
Transform for Classifying Human Emotions Using EEG Signals,” J. Med. Biol. Eng., vol. 31, no.
, pp. 45–51, 2011.
X. Wang, D. Nie, and B. Lu, “EEG-Based Emotion Recognition Using Frequency Domain
Features and Support Vector Machines,” Neural Inf. Process., pp. 734–743, 2011.
F. Ringeval, A. Sonderegger, B. Noris, A. Billard, J. Sauer, and D. Lalanne, “On the Influence of
Emotional Feedback on Emotion Awareness and Gaze Behavior,” 2013 Hum. Assoc. Conf. Affect.
Comput. Intell. Interact., pp. 448–453, Sep. 2013.
V. Kolodyazhniy, S. D. Kreibig, J. J. Gross, W. T. Roth, and F. H. Wilhelm, “An affective
computing approach to physiological emotion specificity: toward subject-independent and
stimulus-independent classification of film-induced emotions.,” Psychophysiology, vol. 48, no. 7,
pp. 908–22, Jul. 2011.
T. F. Bastos-Filho, A. Ferreira, A. C. Atencio, S. Arjunan, and D. Kumar, “Evaluation of Feature
Extraction Techniques in Emotional State Recognition,” 2012 4th Int. Conf. Intell. Hum. Comput.
Interact., pp. 1–6, Dec. 2012.
S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I.
(Yiannis) Patras, “DEAP : A Database for Emotion Analysis Using Physiological Signals,” IEEE
Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31, 2012.
D. Nie, X. Wang, L. Shi, and B. Lu, “EEG-based Emotion Recognition during Watching Movies,”
Proc. 5th Int. IEEE EMBS Conf. Neural Eng., pp. 667–670, 2011.
S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J.-S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I.
Patras, “Single trial classification of EEG and peripheral physiological signals for recognition of
emotions induced by music videos,” Brain informatics, pp. 89–100, 2010.
C. Katsimerou, J. Redi, and I. Heynderickx, “A Computational Model for Mood Recognition,”
nd Int. Conf. UMAP 2014, Aalborg, Denmark, vol. 8538, pp. 122–133, 2014.
J. Russell, “A circumplex model of affect.,” J. Pers. Soc. Psychol., 1980.
L. S. S. Bialoskorski, J. H. D. . Westerink, and E. L. van den Broek, “Mood Swings: An affective
Interactive Art System,” ICST Inst. Comput. Sci. Soc. Informatics Telecommun. Eng. 2009, pp.
–186, 2009.
R. Khosrowabadi, H. C. Quek, A. Wahab, and K. K. Ang, “EEG-based Emotion Recognition
Using Self-Organizing Map for Boundary Detection,” 2010 20th Int. Conf. Pattern Recognit., pp.
–4245, Aug. 2010.
Emotiv, “Emotiv Epoc,” 2014. [Online]. Available: http://www.emotiv.com/epoc.php. [Accessed:
-Jun-2015].
N. Kamaruddin and A. Wahab, “Human behavior state profile mapping based on recalibrated
speech affective space model.,” Conf. Proc. 34th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.
IEEE Eng. Med. Biol. Soc., pp. 2021–4, Jan. 2012.
D. Handayani, H. Yaacob, A. W. A. Rahman, W. Sediono, and A. Shah, “Computational
Modeling of Mood from Sequence of Emotions,” 3rd Int. Conf. Internet Serv. Technol. Inf. Eng.
, pp. 154–158, Feb. 2015.
Y. Guoliang and W. Zhiliang, “Affective computing model based on emotional psychology,”
Affect. Comput. Model Based Emot. Psychol., pp. 251–260, 2006.
M. Bradley and P. J. Lang, “Measuring Emotion: The Self-Assessment Manikin and The Semantic
Differential,” J. Behav. Ther. Exp. Psychiat., vol. 25, no. I, 1994.
K. Šušmáková, “Human Sleep and Sleep EEG Institute of Measurement Science , Slovak
Academy of Sciences,” vol. 4, pp. 4–5, 2004.
B. T. Jap, S. Lal, P. Fischer, and E. Bekiaris, “Using EEG spectral components to assess
algorithms for detecting fatigue,” Expert Syst. Appl., vol. 36, no. 2, pp. 2352–2359, Mar. 2009.
P. Sauseng, B. Griesmayr, R. Freunberger, and W. Klimesch, “Control mechanisms in working
memory: a possible function of EEG theta oscillations.,” Neurosci. Biobehav. Rev., vol. 34, no. 7,
pp. 1015–22, Jun. 2010.
B. Zoefel, R. J. Huster, and C. S. Herrmann, “Neurofeedback training of the upper alpha
frequency band in EEG improves cognitive performance.,” Neuroimage, vol. 54, no. 2, pp. 1427–
, Jan. 2011.
A. Fink, B. Graif, and A. C. Neubauer, “Brain Correlates Underlying Creative Thinking: EEG
Alpha Activity in Professional vs. Novice Dancers.,” Neuroimage, vol. 46, no. 3, pp. 854–62, Jul.
J. Kamiński, A. Brzezicka, M. Gola, and A. Wróbel, “Β Band Oscillations Engagement in Human
Alertness Process.,” Int. J. Psychophysiol., vol. 85, no. 1, pp. 125–8, Jul. 2012.
B. Penolazzi, C. Spironelli, C. Vio, and A. Angrilli, “Brain plasticity in developmental dyslexia
after phonological treatment: a beta EEG band study.,” Behav. Brain Res., vol. 209, no. 1, pp.
–82, May 2010.
