Dynamic continuous emotion recognition method based on electroencephalography and eye movement signals_Journal of Biomedical Engineering

Authors：

ZOU Yangmeng ^1,2 ,  JIE Lilin ^1,2 , WANG Mingxun ³ , LIU Yong ^1,2 , LI Junhua ¹

1. Jiangxi Provincial Key Laboratory of Image Processing and Pattern Recognition, Nanchang Hangkong University, Nanchang 330063, P. R. China;
2. College of Instrument Science and Optoelectronic Engineering, Nanchang Hangkong University, Nanchang 330063, P. R. China;
3. School of Aviation Services and Music, Nanchang Hangkong University, Nanchang 330063, P. R. China;

Corresponding author：

JIE Lilin, Email: jielilin@nchu.edu.cn

Keywords：

Dynamic continuous emotion; Multimodal feature fusion; Electroencephalography signal; Eye movement; Emotion recognition model

DOI：

10.7507/1001-5515.202408013

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Existing emotion recognition research is typically limited to static laboratory settings and has not fully handle the changes in emotional states in dynamic scenarios. To address this problem, this paper proposes a method for dynamic continuous emotion recognition based on electroencephalography (EEG) and eye movement signals. Firstly, an experimental paradigm was designed to cover six dynamic emotion transition scenarios including happy to calm, calm to happy, sad to calm, calm to sad, nervous to calm, and calm to nervous. EEG and eye movement data were collected simultaneously from 20 subjects to fill the gap in current multimodal dynamic continuous emotion datasets. In the valence-arousal two-dimensional space, emotion ratings for stimulus videos were performed every five seconds on a scale of 1 to 9, and dynamic continuous emotion labels were normalized. Subsequently, frequency band features were extracted from the preprocessed EEG and eye movement data. A cascade feature fusion approach was used to effectively combine EEG and eye movement features, generating an information-rich multimodal feature vector. This feature vector was input into four regression models including support vector regression with radial basis function kernel, decision tree, random forest, and K-nearest neighbors, to develop the dynamic continuous emotion recognition model. The results showed that the proposed method achieved the lowest mean square error for valence and arousal across the six dynamic continuous emotions. This approach can accurately recognize various emotion transitions in dynamic situations, offering higher accuracy and robustness compared to using either EEG or eye movement signals alone, making it well-suited for practical applications.

1.	Phelps E A. Emotion and cognition: insights from studies of the human amygdala. Annual Review of Psychology, 2006, 57: 27-53.
2.	Koelstra S, Mühl C, Soleymani M, et al. DEAP: a database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 2011, 3(1): 18-31.
3.	Katsigiannis S, Ramzan N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics, 2018, 22(1): 98-107.
4.	Cheng J, Chen M, Li C, et al. Emotion recognition from multi-channel EEG via deep forest. IEEE Journal of Biomedical and Health Informatics, 2021, 25(2): 453-464.
5.	Liu Y, Ding Y, Li C, et al. Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Computers in Biology and Medicine, 2020, 123: 103927.
6.	Liu D, Dai W, Zhang H, et al. Brain-machine coupled learning method for facial emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10703-10717.
7.	Li W, Xue J, Tan R, et al. Global-local-feature-fused driver speech emotion detection for intelligent cockpit in automated driving. IEEE Transactions on Intelligent Vehicles, 2023, 8(4): 2684-2697.
8.	蔡梓良, 郭苗苗, 杨新生, 等. 基于最大分类器差异域对抗方法的跨受试者脑电情绪识别研究. 生物医学工程学杂志, 2021, 38(3): 455-462.
9.	Chowdary M K, Nguyen T N, Hemanth D J. Deep learning-based facial emotion recognition for human-computer interaction applications. Neural Computing and Applications, 2021, 35(32): 23311-23328.
10.	Fei Z, Yang E, Li D D, et al. Deep convolution network based emotion analysis towards mental health care. Neurocomputing, 2020, 388: 212-227.
11.	Rouast P V, Adam M T P, Chiong R. Deep learning for human affect recognition: insights and new developments. IEEE Transactions on Affective Computing, 2021, 12(2): 524-543.
12.	Yu X, Li Z, Zang Z, et al. Real-time EEG-based emotion recognition. Sensors, 2023, 23(18): 7853.
13.	Duan D, Li Q, Zhong W, et al. GSCNN: gender-sensitive EEG emotion recognition using convolutional neural network//2023 29th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Queenstown: IEEE, 2023: 1-6.
14.	Zhang T, El Ali A, Wang C, et al. Weakly supervised learning for fine-grained emotion recognition using physiological signals. IEEE Transactions on Affective Computing, 2023, 14(3): 2304-2322.
15.	Zhang L, Xiao D, Guo X, et al. Cross-subject emotion EEG signal recognition based on source microstate analysis. Frontiers in Neuroscience, 2023, 17: 1288580.
16.	Li D, Xie L, Wang Z, et al. Brain emotion perception inspired EEG emotion recognition with deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(9): 12979-12992.
17.	Zhang J, Park S, Cho A, et al. Recognition of emotion by brain connectivity and eye movement. Sensors, 2022, 22(18): 6736.
18.	Zheng W L, Liu W, Lu Y, et al. EmotionMeter: a multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, 2019, 49(3): 1110-1122.
19.	Wu X, Zheng W L, Li Z, et al. Investigating EEG-based functional connectivity patterns for multimodal emotion recognition. Journal of Neural Engineering, 2022, 19(1): 016012.
20.	Liu W, Zheng W, Lu B. Emotion recognition using multimodal deep learning//Proceeding of the 23rd International Conference on Neural Information Processing, Kyoto: Springer Cham, 2016: 521-529.
21.	Zhao L, Li R, Zheng W, et al. Classification of five emotions from EEG and eye movement signals: complementary representation properties//2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco: IEEE, 2019: 611-614.
22.	Russell J A. A circumplex model of affect. Journal of Personality & Social Psychology, 1980, 39(6): 1161-1178.
23.	Eysenck S B, Eysenck H J, Barrett P. A revised version of the psychoticism scale. Personality and Individual Differences, 1985, 6(1): 21-29.
24.	Abenna S, Nahid M, Bouyghf H, et al. EEG-based BCI: a novel improvement for EEG signals classification based on real-time preprocessing. Computers in Biology and Medicine, 2022, 148: 105931.
25.	Yan W, He B, Zhao J, et al. Frequency domain filtering method for SSVEP-EEG preprocessing. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 2079-2089.
26.	Aljanaki A, Yang Y H, Soleymani M. Developing a benchmark for emotional analysis of music. PLoS One, 2017, 12(3): e0173392.
27.	杨文阳, 徐可欣. 联合虚拟现实环境和脑电信号的情绪识别研究进展. 生物医学工程学杂志, 2024, 41(2): 389-397.
28.	Zhang Y, Liu H, Zhang D, et al. EEG-based emotion recognition with emotion localization via hierarchical self-attention. IEEE Transactions on Affective Computing, 2023, 14(3): 2458-2469.
29.	汪佳衡, 王跃明, 姚林. 基于滤波器组长短时记忆网络的脑电信号情绪识别. 生物医学工程学杂志, 2021, 38(3): 447-454.
30.	Fu B, Gu C, Fu M, et al. A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals. Frontiers in Neuroscience, 2023, 17: 1234162.
31.	Gong X, Chen C L P, Zhang T. Cross-cultural emotion recognition with EEG and eye movement signals based on multiple stacked broad learning system. IEEE Transactions on Computational Social Systems, 2023, 11(2): 2014-2025.
32.	Li J, Chen N, Zhu H, et al. Incongruity-aware multimodal physiology signals fusion for emotion recognition. Information Fusion, 2024, 105: 102220.
33.	Zhang F, Li X, Lim C P, et al. Deep emotional arousal network for multimodal sentiment analysis and emotion recognition. Information Fusion, 2022, 88: 296-304.

1. Phelps E A. Emotion and cognition: insights from studies of the human amygdala. Annual Review of Psychology, 2006, 57: 27-53.
2. Koelstra S, Mühl C, Soleymani M, et al. DEAP: a database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 2011, 3(1): 18-31.
3. Katsigiannis S, Ramzan N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics, 2018, 22(1): 98-107.
4. Cheng J, Chen M, Li C, et al. Emotion recognition from multi-channel EEG via deep forest. IEEE Journal of Biomedical and Health Informatics, 2021, 25(2): 453-464.
5. Liu Y, Ding Y, Li C, et al. Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Computers in Biology and Medicine, 2020, 123: 103927.
6. Liu D, Dai W, Zhang H, et al. Brain-machine coupled learning method for facial emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10703-10717.
7. Li W, Xue J, Tan R, et al. Global-local-feature-fused driver speech emotion detection for intelligent cockpit in automated driving. IEEE Transactions on Intelligent Vehicles, 2023, 8(4): 2684-2697.
8. 蔡梓良, 郭苗苗, 杨新生, 等. 基于最大分类器差异域对抗方法的跨受试者脑电情绪识别研究. 生物医学工程学杂志, 2021, 38(3): 455-462.
9. Chowdary M K, Nguyen T N, Hemanth D J. Deep learning-based facial emotion recognition for human-computer interaction applications. Neural Computing and Applications, 2021, 35(32): 23311-23328.
10. Fei Z, Yang E, Li D D, et al. Deep convolution network based emotion analysis towards mental health care. Neurocomputing, 2020, 388: 212-227.
11. Rouast P V, Adam M T P, Chiong R. Deep learning for human affect recognition: insights and new developments. IEEE Transactions on Affective Computing, 2021, 12(2): 524-543.
12. Yu X, Li Z, Zang Z, et al. Real-time EEG-based emotion recognition. Sensors, 2023, 23(18): 7853.
13. Duan D, Li Q, Zhong W, et al. GSCNN: gender-sensitive EEG emotion recognition using convolutional neural network//2023 29th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Queenstown: IEEE, 2023: 1-6.
14. Zhang T, El Ali A, Wang C, et al. Weakly supervised learning for fine-grained emotion recognition using physiological signals. IEEE Transactions on Affective Computing, 2023, 14(3): 2304-2322.
15. Zhang L, Xiao D, Guo X, et al. Cross-subject emotion EEG signal recognition based on source microstate analysis. Frontiers in Neuroscience, 2023, 17: 1288580.
16. Li D, Xie L, Wang Z, et al. Brain emotion perception inspired EEG emotion recognition with deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(9): 12979-12992.
17. Zhang J, Park S, Cho A, et al. Recognition of emotion by brain connectivity and eye movement. Sensors, 2022, 22(18): 6736.
18. Zheng W L, Liu W, Lu Y, et al. EmotionMeter: a multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, 2019, 49(3): 1110-1122.
19. Wu X, Zheng W L, Li Z, et al. Investigating EEG-based functional connectivity patterns for multimodal emotion recognition. Journal of Neural Engineering, 2022, 19(1): 016012.
20. Liu W, Zheng W, Lu B. Emotion recognition using multimodal deep learning//Proceeding of the 23rd International Conference on Neural Information Processing, Kyoto: Springer Cham, 2016: 521-529.
21. Zhao L, Li R, Zheng W, et al. Classification of five emotions from EEG and eye movement signals: complementary representation properties//2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco: IEEE, 2019: 611-614.
22. Russell J A. A circumplex model of affect. Journal of Personality & Social Psychology, 1980, 39(6): 1161-1178.
23. Eysenck S B, Eysenck H J, Barrett P. A revised version of the psychoticism scale. Personality and Individual Differences, 1985, 6(1): 21-29.
24. Abenna S, Nahid M, Bouyghf H, et al. EEG-based BCI: a novel improvement for EEG signals classification based on real-time preprocessing. Computers in Biology and Medicine, 2022, 148: 105931.
25. Yan W, He B, Zhao J, et al. Frequency domain filtering method for SSVEP-EEG preprocessing. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 2079-2089.
26. Aljanaki A, Yang Y H, Soleymani M. Developing a benchmark for emotional analysis of music. PLoS One, 2017, 12(3): e0173392.
27. 杨文阳, 徐可欣. 联合虚拟现实环境和脑电信号的情绪识别研究进展. 生物医学工程学杂志, 2024, 41(2): 389-397.
28. Zhang Y, Liu H, Zhang D, et al. EEG-based emotion recognition with emotion localization via hierarchical self-attention. IEEE Transactions on Affective Computing, 2023, 14(3): 2458-2469.
29. 汪佳衡, 王跃明, 姚林. 基于滤波器组长短时记忆网络的脑电信号情绪识别. 生物医学工程学杂志, 2021, 38(3): 447-454.
30. Fu B, Gu C, Fu M, et al. A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals. Frontiers in Neuroscience, 2023, 17: 1234162.
31. Gong X, Chen C L P, Zhang T. Cross-cultural emotion recognition with EEG and eye movement signals based on multiple stacked broad learning system. IEEE Transactions on Computational Social Systems, 2023, 11(2): 2014-2025.
32. Li J, Chen N, Zhu H, et al. Incongruity-aware multimodal physiology signals fusion for emotion recognition. Information Fusion, 2024, 105: 102220.
33. Zhang F, Li X, Lim C P, et al. Deep emotional arousal network for multimodal sentiment analysis and emotion recognition. Information Fusion, 2022, 88: 296-304.

Journal of Biomedical Engineering

Latest ArticlesDynamic continuous emotion recognition method based on electroencephalography and eye movement signals

Abstract Full text Figures/Tables Video References Cited by

Format

Content