Application of multi-scale spatiotemporal networks in physiological signal and facial action unit measurement_Journal of Biomedical Engineering

Authors：

XU Siyuan ,  ZHANG Sunjie

School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, P. R. China;

Corresponding author：

ZHANG Sunjie, Email: zhang_sunjie@126.com

Keywords：

Multi-task learning; Remote photoplethysmography signal measurement; Facial action units

DOI：

10.7507/1001-5515.202408026

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Multi-task learning (MTL) has demonstrated significant advantages in the field of physiological signal measurement. This approach enhances the model's generalization ability by sharing parameters and features between similar tasks, even in data-scarce environments. However, traditional multi-task physiological signal measurement methods face challenges such as feature conflicts between tasks, task imbalance, and excessive model complexity, which limit their application in complex environments. To address these issues, this paper proposes an enhanced multi-scale spatiotemporal network (EMSTN) based on Eulerian video magnification (EVM), super-resolution reconstruction and convolutional multilayer perceptron. First, EVM is introduced in the input stage of the network to amplify subtle color and motion changes in the video, significantly improving the model's ability to capture pulse and respiratory signals. Additionally, a super-resolution reconstruction module is integrated into the network to enhance the image resolution, thereby improving detail capture and increasing the accuracy of facial action unit (AU) tasks. Then, convolutional multilayer perceptron is employed to replace traditional 2D convolutions, improving feature extraction efficiency and flexibility, which significantly boosts the performance of heart rate and respiratory rate measurements. Finally, comprehensive experiments on the Binghamton-Pittsburgh 4D Spontaneous Facial Expression Database (BP4D+) fully validate the effectiveness and superiority of the proposed method in multi-task physiological signal measurement.

1.	Prathosh A P, Praveena P, Mestha L K, et al. Estimation of respiratory pattern from video using selective ensemble aggregation. IEEE Trans Signal Process, 2017, 65(11): 2902-2916.
2.	Song R, Zhang S, Cheng J, et al. New insights on super-high resolution for video-based heart rate estimation with a semi-blind source separation method. Comput Biol Med, 2020, 116: 103535.
3.	Jeon H J, Kim J D, Lim S J. Pulse detection from PPG signal with motion artifact using independent component analysis and nonlinear auto-correlation. J Sens Sci Technol, 2016, 25(1): 71-78.
4.	曾梓琳, 胡志刚, 尚鹏, 等. 基于面部视频的非接触式心率检测方法研究. 计算机科学, 2022, 49(11A): 211000182.
5.	Cipolla R, Gal Y, Kendall A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City: IEEE, 2018: 7482-7491.
6.	Sener O, Koltun V. Multi-task learning as multi-objective optimization. Adv Neural Inf Process Syst, 2018: 31.
7.	Yu T, Kumar S, Gupta A, et al. Gradient surgery for multi-task learning. Adv Neural Inf Process Syst, 2020, 33: 5824-5836.
8.	Caruana R. Multitask learning. Mach Learn, 1997, 28: 41-75.
9.	Kokkinos I. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017: 6129-6138.
10.	Zhang K, Zhang Z, Li Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett, 2016, 23(10): 1499-1503.
11.	刘正成. 基于人脸视频的非接触式心率检测方法研究. 长春: 东北师范大学, 2021.
12.	Ekman P, Levenson R W, Friesen W V. Autonomic nervous system activity distinguishes among emotions. Science, 1983, 221(4616): 1208-1210.
13.	Martinez B, Valstar M F, Jiang B, et al. Automatic analysis of facial actions: a survey. IEEE Trans Affective Comput, 2017, 10(3): 325-347.
14.	McDuff D. Camera measurement of physiological vital signs. ACM Comput Surv, 2023, 55(9): 176.
15.	Verkruysse W, Svaasand L O, Nelson J S. Remote plethysmographic imaging using ambient light. Opt Express, 2008, 16(26): 21434-21445.
16.	Liu X, Fromm J, Patel S, et al. Multi-task temporal shift attention networks for on-device contactless vitals measurement. Adv Neural Inf Process Syst, 2020, 33: 19400-19411.
17.	Narayanswamy G, Liu Y, Yang Y, et al. BigSmall: efficient multi-task learning for disparate spatial and temporal physiological measurements//2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa: IEEE, 2024: 7914-7924.
18.	Lin J, Gan C, Han S. TSM: Temporal shift module for efficient video understanding//2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019: 7083-7093.
19.	Qiu Y, Liu Y, Arteaga-Falconi J, et al. EVM-CNN: real-time contactless heart rate estimation from facial video. IEEE Trans Multimedia, 2019, 21(7): 1778-1787.
20.	Jaiswal K B, Meenpal T. rPPG-FuseNet: non-contact heart rate estimation from facial video via RGB/MSR signal fusion. Biomed Signal Process Control, 2022, 78: 104002.
21.	Zhang J, Yan B, Du X, et al. Motion magnification multi-feature relation network for facial microexpression recognition. Complex Intell Syst, 2022, 8(4): 3363-3376.
22.	Dong C, Loy C C, He K, et al. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell, 2015, 38(2): 295-307.
23.	Cao M, Liu Z, Huang X, et al. Research for face image super-resolution reconstruction based on wavelet transform and SRGAN//2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing: IEEE, 2021, 5: 448-451.
24.	Shi W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 1874-1883.
25.	Zhang Z, Girard J M, Wu Y, et al. Multimodal spontaneous emotion corpus for human behavior analysis//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 3438-3446.
26.	Wang W, den Brinker A C, Stuijk S, et al. Algorithmic principles of remote PPG. IEEE Trans Biomed Eng, 2016, 64(7): 1479-1491.
27.	de Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng, 2013, 60(10): 2878-2886.
28.	Liu X, Hill B, Jiang Z, et al. EfficientPhys: enabling simple, fast and accurate camera-based cardiac measurement//2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa: IEEE, 2023: 5008-5017.
29.	Chen W, McDuff D. DeepPhys: Video-based physiological measurement using convolutional attention networks//Proceedings of the European Conference on Computer Vision (ECCV), Springer, 2018: 349-365.
30.	Zhao K, Chu W S, Zhang H. Deep region and multi-label learning for facial action unit detection//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 3391-3399.
31.	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84-90.

1. Prathosh A P, Praveena P, Mestha L K, et al. Estimation of respiratory pattern from video using selective ensemble aggregation. IEEE Trans Signal Process, 2017, 65(11): 2902-2916.
2. Song R, Zhang S, Cheng J, et al. New insights on super-high resolution for video-based heart rate estimation with a semi-blind source separation method. Comput Biol Med, 2020, 116: 103535.
3. Jeon H J, Kim J D, Lim S J. Pulse detection from PPG signal with motion artifact using independent component analysis and nonlinear auto-correlation. J Sens Sci Technol, 2016, 25(1): 71-78.
4. 曾梓琳, 胡志刚, 尚鹏, 等. 基于面部视频的非接触式心率检测方法研究. 计算机科学, 2022, 49(11A): 211000182.
5. Cipolla R, Gal Y, Kendall A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City: IEEE, 2018: 7482-7491.
6. Sener O, Koltun V. Multi-task learning as multi-objective optimization. Adv Neural Inf Process Syst, 2018: 31.
7. Yu T, Kumar S, Gupta A, et al. Gradient surgery for multi-task learning. Adv Neural Inf Process Syst, 2020, 33: 5824-5836.
8. Caruana R. Multitask learning. Mach Learn, 1997, 28: 41-75.
9. Kokkinos I. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017: 6129-6138.
10. Zhang K, Zhang Z, Li Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett, 2016, 23(10): 1499-1503.
11. 刘正成. 基于人脸视频的非接触式心率检测方法研究. 长春: 东北师范大学, 2021.
12. Ekman P, Levenson R W, Friesen W V. Autonomic nervous system activity distinguishes among emotions. Science, 1983, 221(4616): 1208-1210.
13. Martinez B, Valstar M F, Jiang B, et al. Automatic analysis of facial actions: a survey. IEEE Trans Affective Comput, 2017, 10(3): 325-347.
14. McDuff D. Camera measurement of physiological vital signs. ACM Comput Surv, 2023, 55(9): 176.
15. Verkruysse W, Svaasand L O, Nelson J S. Remote plethysmographic imaging using ambient light. Opt Express, 2008, 16(26): 21434-21445.
16. Liu X, Fromm J, Patel S, et al. Multi-task temporal shift attention networks for on-device contactless vitals measurement. Adv Neural Inf Process Syst, 2020, 33: 19400-19411.
17. Narayanswamy G, Liu Y, Yang Y, et al. BigSmall: efficient multi-task learning for disparate spatial and temporal physiological measurements//2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa: IEEE, 2024: 7914-7924.
18. Lin J, Gan C, Han S. TSM: Temporal shift module for efficient video understanding//2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019: 7083-7093.
19. Qiu Y, Liu Y, Arteaga-Falconi J, et al. EVM-CNN: real-time contactless heart rate estimation from facial video. IEEE Trans Multimedia, 2019, 21(7): 1778-1787.
20. Jaiswal K B, Meenpal T. rPPG-FuseNet: non-contact heart rate estimation from facial video via RGB/MSR signal fusion. Biomed Signal Process Control, 2022, 78: 104002.
21. Zhang J, Yan B, Du X, et al. Motion magnification multi-feature relation network for facial microexpression recognition. Complex Intell Syst, 2022, 8(4): 3363-3376.
22. Dong C, Loy C C, He K, et al. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell, 2015, 38(2): 295-307.
23. Cao M, Liu Z, Huang X, et al. Research for face image super-resolution reconstruction based on wavelet transform and SRGAN//2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing: IEEE, 2021, 5: 448-451.
24. Shi W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 1874-1883.
25. Zhang Z, Girard J M, Wu Y, et al. Multimodal spontaneous emotion corpus for human behavior analysis//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 3438-3446.
26. Wang W, den Brinker A C, Stuijk S, et al. Algorithmic principles of remote PPG. IEEE Trans Biomed Eng, 2016, 64(7): 1479-1491.
27. de Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng, 2013, 60(10): 2878-2886.
28. Liu X, Hill B, Jiang Z, et al. EfficientPhys: enabling simple, fast and accurate camera-based cardiac measurement//2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa: IEEE, 2023: 5008-5017.
29. Chen W, McDuff D. DeepPhys: Video-based physiological measurement using convolutional attention networks//Proceedings of the European Conference on Computer Vision (ECCV), Springer, 2018: 349-365.
30. Zhao K, Chu W S, Zhang H. Deep region and multi-label learning for facial action unit detection//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 3391-3399.
31. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84-90.

Journal of Biomedical Engineering

Latest ArticlesApplication of multi-scale spatiotemporal networks in physiological signal and facial action unit measurement

Abstract Full text Figures/Tables Video References Cited by

Format

Content