A review of deep learning methods for non-contact heart rate measurement based on facial videos_Journal of Biomedical Engineering

Authors：

GUAN Shuyue ¹ , LYU Yimou ¹ , LI Yongchun ² , XIA Chengzhi ¹ ,  QI Lin ^1,3,4 , XU Lisheng ^1,3,4

1. College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110100, P. R. China;
2. Shenyang Kanghui Brain Intelligence Collaborative Innovation Center, Shenyang 110100, P. R. China;
3. Key Laboratory of Medical Image Computing, Ministry of Education, Northeastern University, Shenyang 110169, P. R. China;
4. Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education, Northeastern University, Shenyang 110169, P. R. China;

Corresponding author：

Keywords：

Remote photoplethysmography (rPPG); Deep learning; Non-contact; Heart rate measurement

DOI：

10.7507/1001-5515.200405057

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Heart rate is a crucial indicator of human health with significant physiological importance. Traditional contact methods for measuring heart rate, such as electrocardiograph or wristbands, may not always meet the need for convenient health monitoring. Remote photoplethysmography (rPPG) provides a non-contact method for measuring heart rate and other physiological indicators by analyzing blood volume pulse signals. This approach is non-invasive, does not require direct contact, and allows for long-term healthcare monitoring. Deep learning has emerged as a powerful tool for processing complex image and video data, and has been increasingly employed to extract heart rate signals remotely. This article reviewed the latest research advancements in rPPG-based heart rate measurement using deep learning, summarized available public datasets, and explored future research directions and potential advancements in non-contact heart rate measurement.

1.	Verkruysse W, Svaasand L O, Nelson J S. Remote plethysmographic imaging using ambient light. Opt Express, 2008, 16(26): 21434-21445.
2.	牛雪松, 韩琥, 山世光. 基于rPPG的生理指标测量方法综述. 中国图象图形学报, 2020, 25(11): 2321-2336.
3.	De Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng, 2013, 60(10): 2878-2886.
4.	Poh M Z, McDuff D J, Picard R W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express, 2010, 18(10): 10762-10774.
5.	Manzone T A, Dam H Q, Soltis D, et al. Blood volume analysis: A new technique and new clinical interest reinvigorate a classic study. J Nucl Med Technol, 2007, 35(2): 55-63.
6.	Balakrishnan G, Durand F, Guttag J V. Detecting pulse from head motions in video// 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Portland: IEEE, 2013: 3430-3437.
7.	Wang W, den Brinker A C, Stuijk S, et al. Algorithmic principles of remote PPG. IEEE Trans Biomed Eng, 2017, 64(7): 1479-1491.
8.	Chen W V, McDuff D J. DeepPhys: Video-based physiological measurement using convolutional attention networks// European conference on computer vision (ECCV). Munich: ECCV, 2018: 349-365.
9.	Jaiswal K B, Meenpal T. rPPG-FuseNet: Non-contact heart rate estimation from facial video via RGB/MSR signal fusion. Biomed Signal Proces, 2022, 78: 104002.
10.	Yu Z, Li X, Niu X, et al. AutoHR: A strong end-to-end baseline for remote heart rate measurement with neural searching. IEEE Signal Process Lett, 2020, 27: 1245-1249.
11.	Niu X, Yu Z, Han H, et al. Video-based remote physiological measurement via cross-verified feature disentangling// 2020 16th European Conference on Computer Vision (ECCV). Glasgow: Springer International, 2020: 295-310.
12.	Perepelkina O S, Artemyev M, Churikova M, et al. HeartTrack: Convolutional neural network for remote video-based heart rate monitoring// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle: IEEE, 2020: 288-289.
13.	Gao Haoyuan, Wu Xiaopei, Geng Jidong, et al. Remote heart rate estimation by signal quality attention network// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). New Orleans: IEEE, 2022: 2121-2128.
14.	Gao H, Zhang C, Pei S, et al. LSTM-based real-time signal quality assessment for blood volume pulse analysis. Biomed Opt Express, 2023, 14(3): 1119-1136.
15.	Bian M, Peng B, Wang W, et al. An accurate LSTM based video heart rate estimation method// Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Xi’an: Springer International, 2019: 409-417.
16.	Yu Z, Shen Y, Shi J, et al. PhysFormer: Facial video-based physiological measurement with temporal difference transformer// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE, 2022: 4176-4186.
17.	Yu Z, Shen Y, Shi J, et al. PhysFormer++: Facial video-based physiological measurement with slowfast temporal difference transformer. Int J Comput Vision, 2023, 131(6): 1307-1330.
18.	Liu L, Xia Z, Zhang X, et al. Information-enhanced network for noncontact heart rate estimation from facial videos. IEEE Trans Circuits Syst Video Technol, 2024, 34: 2136-2150.
19.	Gupta A K, Kumar R, Birla L, et al. RADIANT: Better rPPG estimation using signal embeddings and Transformer// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa: IEEE, 2023: 4965-4975.
20.	Du J, Liu S, Zhang B, et al. Weakly supervised rPPG estimation for respiratory rate estimation// 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Montrea: IEEE, 2021: 2391-2397.
21.	Li B, Zhang W, Li X, et al. ECG signal reconstruction based on facial videos via combined explicit and implicit supervision. Knowl Based Syst, 2023, 272: 110608.
22.	Hasan Z, Md Faridee A Z, Ahmed M, et al. SrPPG: Semi-supervised adversarial learning for remote photoplethysmography with noisy data// 2023 IEEE International Conference on Smart Computing (SMARTCOMP). Nashville: IEEE, 2023: 25-32.
23.	Wang R X, Sun H-M, Hao R R, et al. TransPhys: Transformer-based unsupervised contrastive learning for remote heart rate measurement. Biomed Signal Proces, 2023, 86: 105058.
24.	Birla L, Shukla S, Gupta A K, et al. ALPINE: Improving remote heart rate estimation using contrastive learning// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa: IEEE, 2023: 5018-5027.
25.	Gideon J, Stent S. The way to my heart is through contrastive learning: Remote photoplethysmography from unlabelled video// 2021 IEEE/CVF International Conference on Computer Vision(ICCV). Cambridge: IEEE, 2021: 3995-4004.
26.	Park S, Kim B K, Dong S Y. Self-supervised RGB-NIR fusion video vision Transformer framework for rPPG estimation. IEEE Trans Instrum Meas, 2022, 71: 1-10.
27.	Speth J, Vance N, Flynn P, et al. Non-contrastive unsupervised learning of physiological signals from video// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver: IEEE, 2023: 14464-14474.
28.	Liu X, Zhang Y, Yu Z, et al. rPPG-MAE: Self-supervised pretraining with masked autoencoders for remote physiological measurements. IEEE T Multimedia, 2024, 26: 7278-7293.
29.	Stricker R, Müller S, Gross H M. Non-contact video-based pulse rate measurement on a mobile service robot// The 23rd IEEE International Symposium on Robot and Human Interactive Communication. Edinburgh: IEEE, 2014: 1056-1062.
30.	Tasli H E, Gudi A, Den Uyl M. Remote PPG based vital sign measurement using adaptive facial regions// 2014 IEEE International Conference on Image Processing (ICIP). Paris: IEEE, 2014: 1410-1414.
31.	Tulyakov S, Alameda Pineda X, Ricci E, et al. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 2396-2404.
32.	Hsu G-S J, Ambikapathi A, Chen M. Deep learning with time-frequency representation for pulse estimation from facial videos// 2017 IEEE International Joint Conference on Biometrics (IJCB). Denver: IEEE, 2017: 383-389.
33.	Heusch G, Anjos A, Marcel S. A reproducible study on remote heart rate measurement. International Workshops and Challenges. arXiv, 2017: 1709.00962.
34.	Niu X, Han H, Shan S, et al. VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video// Asian Conference on Computer Vision(ACCV). Perth: Springer International, 2018: 562-576.
35.	Spetlik R, Franc V, Cech J, et al. Visual heart rate estimation with convolutional neural network// British Machine Vision Conference (BMVC). Newcastle: BMVA, 2018: 3-6.
36.	Li X, Alikhani I, Shi J, et al. The OBF Database: A large face video database for remote physiological signal measurement and atrial fibrillation detection// 13th IEEE International Conference on Automatic Face & Gesture Recognition. Xi’an: IEEE, 2018: 242-249.
37.	Bobbia S, Macwan R, Benezeth Y, et al. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit Lett, 2017, 124: 82-90.
38.	Pilz C S, Zaunseder S, Krajewski J, et al. Local group invariance for heart rate estimation from face videos in the wild// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City: IEEE, 2018: 1254-1262.
39.	Huang B, Chen W, Lin C-L, et al. A neonatal dataset and benchmark for non-contact neonatal heart rate monitoring based on spatio-temporal neural networks. Eng Appl Artif Intell, 2021, 106: 104447.
40.	Tang J, Chen K, Wang Y, et al. MMPD: multi-domain mobile video physiology dataset// 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Sydney: IEEE, 2023: 1-5.
41.	Revanur A, Li Z, Ciftci U A, et al. The first vision for vitals (V4V) challenge for non-contact video-based physiological estimation// 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Montreal: IEEE, 2021: 2760-2767.
42.	McDuff D J, Wander M, Liu X, et al. SCAMPS: Synthetics for camera measurement of physiological signals// 36th International Conference on Neural Information Processing Systems. New Orleans: Curran Associates Inc, 2022: 3744-3757.
43.	Magdalena Nowara E, Marks T K, Mansour H, et al. SparsePPG: Towards driver monitoring using camera-based vital signs estimation in near-infrared// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City: IEEE, 2018: 1272-1281.
44.	Nowara E M, Marks T K, Mansour H, et al. Near-infrared imaging photoplethysmography during driving. IEEE T Intell Transp, 2020, 23(4): 3589-3600.
45.	Koelstra S, Mühl C, Soleymani M, et al. DEAP: A database for emotion analysis using physiological signals. IEEE T Affect Comput, 2012, 3: 18-31.
46.	Soleymani M, Lichtenauer J, Pun T, et al. A multimodal database for affect recognition and implicit tagging. IEEE T Affect Comput, 2011, 3(1): 42-55.
47.	Chen S, Wong K L, Chan T T, et al. An image enhancement based method for improving rPPG extraction under low-light illumination. Biomed Signal Proces, 2025, 100: 106963.

1. Verkruysse W, Svaasand L O, Nelson J S. Remote plethysmographic imaging using ambient light. Opt Express, 2008, 16(26): 21434-21445.
2. 牛雪松, 韩琥, 山世光. 基于rPPG的生理指标测量方法综述. 中国图象图形学报, 2020, 25(11): 2321-2336.
3. De Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng, 2013, 60(10): 2878-2886.
4. Poh M Z, McDuff D J, Picard R W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express, 2010, 18(10): 10762-10774.
5. Manzone T A, Dam H Q, Soltis D, et al. Blood volume analysis: A new technique and new clinical interest reinvigorate a classic study. J Nucl Med Technol, 2007, 35(2): 55-63.
6. Balakrishnan G, Durand F, Guttag J V. Detecting pulse from head motions in video// 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Portland: IEEE, 2013: 3430-3437.
7. Wang W, den Brinker A C, Stuijk S, et al. Algorithmic principles of remote PPG. IEEE Trans Biomed Eng, 2017, 64(7): 1479-1491.
8. Chen W V, McDuff D J. DeepPhys: Video-based physiological measurement using convolutional attention networks// European conference on computer vision (ECCV). Munich: ECCV, 2018: 349-365.
9. Jaiswal K B, Meenpal T. rPPG-FuseNet: Non-contact heart rate estimation from facial video via RGB/MSR signal fusion. Biomed Signal Proces, 2022, 78: 104002.
10. Yu Z, Li X, Niu X, et al. AutoHR: A strong end-to-end baseline for remote heart rate measurement with neural searching. IEEE Signal Process Lett, 2020, 27: 1245-1249.
11. Niu X, Yu Z, Han H, et al. Video-based remote physiological measurement via cross-verified feature disentangling// 2020 16th European Conference on Computer Vision (ECCV). Glasgow: Springer International, 2020: 295-310.
12. Perepelkina O S, Artemyev M, Churikova M, et al. HeartTrack: Convolutional neural network for remote video-based heart rate monitoring// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle: IEEE, 2020: 288-289.
13. Gao Haoyuan, Wu Xiaopei, Geng Jidong, et al. Remote heart rate estimation by signal quality attention network// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). New Orleans: IEEE, 2022: 2121-2128.
14. Gao H, Zhang C, Pei S, et al. LSTM-based real-time signal quality assessment for blood volume pulse analysis. Biomed Opt Express, 2023, 14(3): 1119-1136.
15. Bian M, Peng B, Wang W, et al. An accurate LSTM based video heart rate estimation method// Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Xi’an: Springer International, 2019: 409-417.
16. Yu Z, Shen Y, Shi J, et al. PhysFormer: Facial video-based physiological measurement with temporal difference transformer// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE, 2022: 4176-4186.
17. Yu Z, Shen Y, Shi J, et al. PhysFormer++: Facial video-based physiological measurement with slowfast temporal difference transformer. Int J Comput Vision, 2023, 131(6): 1307-1330.
18. Liu L, Xia Z, Zhang X, et al. Information-enhanced network for noncontact heart rate estimation from facial videos. IEEE Trans Circuits Syst Video Technol, 2024, 34: 2136-2150.
19. Gupta A K, Kumar R, Birla L, et al. RADIANT: Better rPPG estimation using signal embeddings and Transformer// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa: IEEE, 2023: 4965-4975.
20. Du J, Liu S, Zhang B, et al. Weakly supervised rPPG estimation for respiratory rate estimation// 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Montrea: IEEE, 2021: 2391-2397.
21. Li B, Zhang W, Li X, et al. ECG signal reconstruction based on facial videos via combined explicit and implicit supervision. Knowl Based Syst, 2023, 272: 110608.
22. Hasan Z, Md Faridee A Z, Ahmed M, et al. SrPPG: Semi-supervised adversarial learning for remote photoplethysmography with noisy data// 2023 IEEE International Conference on Smart Computing (SMARTCOMP). Nashville: IEEE, 2023: 25-32.
23. Wang R X, Sun H-M, Hao R R, et al. TransPhys: Transformer-based unsupervised contrastive learning for remote heart rate measurement. Biomed Signal Proces, 2023, 86: 105058.
24. Birla L, Shukla S, Gupta A K, et al. ALPINE: Improving remote heart rate estimation using contrastive learning// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa: IEEE, 2023: 5018-5027.
25. Gideon J, Stent S. The way to my heart is through contrastive learning: Remote photoplethysmography from unlabelled video// 2021 IEEE/CVF International Conference on Computer Vision(ICCV). Cambridge: IEEE, 2021: 3995-4004.
26. Park S, Kim B K, Dong S Y. Self-supervised RGB-NIR fusion video vision Transformer framework for rPPG estimation. IEEE Trans Instrum Meas, 2022, 71: 1-10.
27. Speth J, Vance N, Flynn P, et al. Non-contrastive unsupervised learning of physiological signals from video// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver: IEEE, 2023: 14464-14474.
28. Liu X, Zhang Y, Yu Z, et al. rPPG-MAE: Self-supervised pretraining with masked autoencoders for remote physiological measurements. IEEE T Multimedia, 2024, 26: 7278-7293.
29. Stricker R, Müller S, Gross H M. Non-contact video-based pulse rate measurement on a mobile service robot// The 23rd IEEE International Symposium on Robot and Human Interactive Communication. Edinburgh: IEEE, 2014: 1056-1062.
30. Tasli H E, Gudi A, Den Uyl M. Remote PPG based vital sign measurement using adaptive facial regions// 2014 IEEE International Conference on Image Processing (ICIP). Paris: IEEE, 2014: 1410-1414.
31. Tulyakov S, Alameda Pineda X, Ricci E, et al. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 2396-2404.
32. Hsu G-S J, Ambikapathi A, Chen M. Deep learning with time-frequency representation for pulse estimation from facial videos// 2017 IEEE International Joint Conference on Biometrics (IJCB). Denver: IEEE, 2017: 383-389.
33. Heusch G, Anjos A, Marcel S. A reproducible study on remote heart rate measurement. International Workshops and Challenges. arXiv, 2017: 1709.00962.
34. Niu X, Han H, Shan S, et al. VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video// Asian Conference on Computer Vision(ACCV). Perth: Springer International, 2018: 562-576.
35. Spetlik R, Franc V, Cech J, et al. Visual heart rate estimation with convolutional neural network// British Machine Vision Conference (BMVC). Newcastle: BMVA, 2018: 3-6.
36. Li X, Alikhani I, Shi J, et al. The OBF Database: A large face video database for remote physiological signal measurement and atrial fibrillation detection// 13th IEEE International Conference on Automatic Face & Gesture Recognition. Xi’an: IEEE, 2018: 242-249.
37. Bobbia S, Macwan R, Benezeth Y, et al. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit Lett, 2017, 124: 82-90.
38. Pilz C S, Zaunseder S, Krajewski J, et al. Local group invariance for heart rate estimation from face videos in the wild// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City: IEEE, 2018: 1254-1262.
39. Huang B, Chen W, Lin C-L, et al. A neonatal dataset and benchmark for non-contact neonatal heart rate monitoring based on spatio-temporal neural networks. Eng Appl Artif Intell, 2021, 106: 104447.
40. Tang J, Chen K, Wang Y, et al. MMPD: multi-domain mobile video physiology dataset// 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Sydney: IEEE, 2023: 1-5.
41. Revanur A, Li Z, Ciftci U A, et al. The first vision for vitals (V4V) challenge for non-contact video-based physiological estimation// 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Montreal: IEEE, 2021: 2760-2767.
42. McDuff D J, Wander M, Liu X, et al. SCAMPS: Synthetics for camera measurement of physiological signals// 36th International Conference on Neural Information Processing Systems. New Orleans: Curran Associates Inc, 2022: 3744-3757.
43. Magdalena Nowara E, Marks T K, Mansour H, et al. SparsePPG: Towards driver monitoring using camera-based vital signs estimation in near-infrared// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City: IEEE, 2018: 1272-1281.
44. Nowara E M, Marks T K, Mansour H, et al. Near-infrared imaging photoplethysmography during driving. IEEE T Intell Transp, 2020, 23(4): 3589-3600.
45. Koelstra S, Mühl C, Soleymani M, et al. DEAP: A database for emotion analysis using physiological signals. IEEE T Affect Comput, 2012, 3: 18-31.
46. Soleymani M, Lichtenauer J, Pun T, et al. A multimodal database for affect recognition and implicit tagging. IEEE T Affect Comput, 2011, 3(1): 42-55.
47. Chen S, Wong K L, Chan T T, et al. An image enhancement based method for improving rPPG extraction under low-light illumination. Biomed Signal Proces, 2025, 100: 106963.

Journal of Biomedical Engineering

Latest ArticlesA review of deep learning methods for non-contact heart rate measurement based on facial videos

Abstract Full text Figures/Tables Video References Cited by

Format

Content