Study on speech imagery electroencephalography decoding of Chinese words based on the CAM-Net model_Journal of Biomedical Engineering

Authors：

LIU Xiaolong ,  YANG Banghua , GAN An’an ,  ZHANG Jie

School of Mechanical and Electrical Engineering and Automation, Shanghai University, Shanghai 200444, P. R. China;

Corresponding author：

YANG Banghua, Email: yangbanghua@shu.edu.cn; ZHANG Jie, Email: jjy2001_cn@shu.edu.cn

Keywords：

Brain-computer interface; Speech imagery; Attention mechanism; Long short-term memory network

DOI：

10.7507/1001-5515.202503048

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Speech imagery is an emerging brain-computer interface (BCI) paradigm with potential to provide effective communication for individuals with speech impairments. This study designed a Chinese speech imagery paradigm using three clinically relevant words—“Help me”, “Sit up” and “Turn over”—and collected electroencephalography (EEG) data from 15 healthy subjects. Based on the data, a Channel Attention Multi-Scale Convolutional Neural Network (CAM-Net) decoding algorithm was proposed, which combined multi-scale temporal convolutions with asymmetric spatial convolutions to extract multidimensional EEG features, and incorporated a channel attention mechanism along with a bidirectional long short-term memory network to perform channel weighting and capture temporal dependencies. Experimental results showed that CAM-Net achieved a classification accuracy of 48.54% in the three-class task, outperforming baseline models such as EEGNet and Deep ConvNet, and reached a highest accuracy of 64.17% in the binary classification between “Sit up” and “Turn over.” This work provides a promising approach for future Chinese speech imagery BCI research and applications.

1.	Vaughan T M, Heetderks W J, Trejo L J, et al. Brain-computer interface technology: a review of the Second International Meeting. IEEE Trans Neural Syst Rehabil Eng, 2003, 11(2): 94-109.
2.	伏云发, 郭衍龙, 张夏冰. 脑-机接口—革命性的人机交互. 北京: 国防工业出版社, 2020.
3.	刘艳鹏, 龚安民, 丁鹏, 等. 基于言语想象的脑机交互关键技术. 生物医学工程学杂志, 2022, 39(3): 596-611.
4.	Anumanchipalli G K, Chartier J, Chang E F. Speech synthesis from neural decoding of spoken sentences. Nature, 2019, 568(7753): 493-498.
5.	Riaz A, Akhtar S, Iftikhar S, et al. Inter comparison of classification techniques for vowel speech imagery using EEG sensors// The 2014 2nd International Conference on Systems and Informatics (ICSAI 2014). Shanghai: IEEE, 2014: 712-717.
6.	Zhang H, Guo Z, Chen F. The effects of different brain regions on fNIRS-based task-state detection in speech imagery// 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Sydney: IEEE, 2023: 1-4.
7.	Panachakel J T, Ramakrishnan A G. Decoding covert speech from EEG-a comprehensive review. Front Neurosci, 2021, 15: 642251.
8.	Van Gerven M, Farquhar J, Schaefer R, et al. The brain–computer interface cycle. J Neural Eng, 2009, 6(4): 041001.
9.	Lee S H, Lee Y E, Lee S W. Toward imagined speech based smart communication system: potential applications on metaverse conditions// 2022 10th International Winter Conference on Brain-Computer Interface (BCI). Gangwon-do: IEEE, 2022: 1-4.
10.	阮盛捷, 谢宏. 基于功能近红外信号的脑机接口(BCI)在肌萎缩性侧索硬化症(ALS)患者交流中的研究. 现代计算机(专业版), 2018(17): 3-7.
11.	Bisla M, Anand R S. Analysis of imagined speech characteristics using phase-based connectivity measures// 2023 IEEE 13th International Conference on Control System, Computing and Engineering (ICCSCE). Penang: IEEE, 2023: 68-73.
12.	郭苗苗, 齐志光, 王磊, 等. 语言脑机接口康复系统中的参数优化研究. 信号处理, 2018, 34(8): 974-983.
13.	Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Mag, 2007, 25(1): 41-56.
14.	Wang X, Lai Y H, Chen F. Intended speech classification with EEG signals based on a temporal attention mechanism: A study of mandarin vowels// 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando: IEEE, 2024: 1-4.
15.	Zhang Z, Ding X, Bao Y, et al. Chisco: An EEG-based BCI dataset for decoding of imagined speech. Sci Data, 2024, 11(1): 1265.
16.	Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng, 2018, 15(5): 056013.
17.	Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
18.	Hermosilla D M, Codorniú R T, Baracaldo R L, et al. Shallow convolutional network excel for classifying motor imagery EEG in BCI applications. IEEE Access, 2021, 9: 98275-98286.
19.	Ding Y, Robinson N, Zhang S, et al. TSception: Capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Trans Affect Comput, 2022, 14(3): 2238-2250.
20.	Krishna G, Tran C, Han Y, et al. Speech synthesis using EEG// ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona: IEEE, 2020: 1235-1238.
21.	Lotte F, Brumberg J S, Brunner P, et al. Electrocorticographic representations of segmental features in continuous speech. Front Hum Neurosci, 2015, 9: 97.
22.	Bocquelet F, Hueber T, Girin L, et al. Key considerations in designing a speech brain-computer interface. J Physiol Paris, 2016, 110(4): 392-401.
23.	Ang K K, Chin Z Y, Zhang H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface// 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). Hong Kong: IEEE, 2008: 2390-2397.

1. Vaughan T M, Heetderks W J, Trejo L J, et al. Brain-computer interface technology: a review of the Second International Meeting. IEEE Trans Neural Syst Rehabil Eng, 2003, 11(2): 94-109.
2. 伏云发, 郭衍龙, 张夏冰. 脑-机接口—革命性的人机交互. 北京: 国防工业出版社, 2020.
3. 刘艳鹏, 龚安民, 丁鹏, 等. 基于言语想象的脑机交互关键技术. 生物医学工程学杂志, 2022, 39(3): 596-611.
4. Anumanchipalli G K, Chartier J, Chang E F. Speech synthesis from neural decoding of spoken sentences. Nature, 2019, 568(7753): 493-498.
5. Riaz A, Akhtar S, Iftikhar S, et al. Inter comparison of classification techniques for vowel speech imagery using EEG sensors// The 2014 2nd International Conference on Systems and Informatics (ICSAI 2014). Shanghai: IEEE, 2014: 712-717.
6. Zhang H, Guo Z, Chen F. The effects of different brain regions on fNIRS-based task-state detection in speech imagery// 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Sydney: IEEE, 2023: 1-4.
7. Panachakel J T, Ramakrishnan A G. Decoding covert speech from EEG-a comprehensive review. Front Neurosci, 2021, 15: 642251.
8. Van Gerven M, Farquhar J, Schaefer R, et al. The brain–computer interface cycle. J Neural Eng, 2009, 6(4): 041001.
9. Lee S H, Lee Y E, Lee S W. Toward imagined speech based smart communication system: potential applications on metaverse conditions// 2022 10th International Winter Conference on Brain-Computer Interface (BCI). Gangwon-do: IEEE, 2022: 1-4.
10. 阮盛捷, 谢宏. 基于功能近红外信号的脑机接口(BCI)在肌萎缩性侧索硬化症(ALS)患者交流中的研究. 现代计算机(专业版), 2018(17): 3-7.
11. Bisla M, Anand R S. Analysis of imagined speech characteristics using phase-based connectivity measures// 2023 IEEE 13th International Conference on Control System, Computing and Engineering (ICCSCE). Penang: IEEE, 2023: 68-73.
12. 郭苗苗, 齐志光, 王磊, 等. 语言脑机接口康复系统中的参数优化研究. 信号处理, 2018, 34(8): 974-983.
13. Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Mag, 2007, 25(1): 41-56.
14. Wang X, Lai Y H, Chen F. Intended speech classification with EEG signals based on a temporal attention mechanism: A study of mandarin vowels// 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando: IEEE, 2024: 1-4.
15. Zhang Z, Ding X, Bao Y, et al. Chisco: An EEG-based BCI dataset for decoding of imagined speech. Sci Data, 2024, 11(1): 1265.
16. Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng, 2018, 15(5): 056013.
17. Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
18. Hermosilla D M, Codorniú R T, Baracaldo R L, et al. Shallow convolutional network excel for classifying motor imagery EEG in BCI applications. IEEE Access, 2021, 9: 98275-98286.
19. Ding Y, Robinson N, Zhang S, et al. TSception: Capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Trans Affect Comput, 2022, 14(3): 2238-2250.
20. Krishna G, Tran C, Han Y, et al. Speech synthesis using EEG// ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona: IEEE, 2020: 1235-1238.
21. Lotte F, Brumberg J S, Brunner P, et al. Electrocorticographic representations of segmental features in continuous speech. Front Hum Neurosci, 2015, 9: 97.
22. Bocquelet F, Hueber T, Girin L, et al. Key considerations in designing a speech brain-computer interface. J Physiol Paris, 2016, 110(4): 392-401.
23. Ang K K, Chin Z Y, Zhang H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface// 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). Hong Kong: IEEE, 2008: 2390-2397.

Journal of Biomedical Engineering

Latest ArticlesStudy on speech imagery electroencephalography decoding of Chinese words based on the CAM-Net model

Abstract Full text Figures/Tables Video References Cited by

Format

Content