Application of large language models in sarcopenia diagnosis and treatment: a comparative study with clinical decision-making by physicians_Chinese Journal of Evidence-Based Medicine

Authors：

MA Zebing ¹ , LIU Yibing ¹ , FAN Huyu ² , YANG Chaofan ² , KONG Liang ² , CHEN Rui ² ,  CAO Xiangyang ^2,3,4,5

1. Hunan University of Chinese Medicine, Changsha 410000, P. R. China;
2. Luoyang Orthopedic Hospital of Henan Province (Orthopedic Hospital of Henan Province), Zhengzhou 450000, P. R. China;
3. .Institute of Intelligent Medical and Bioengineering, Henan Academy of Traditional Chinese Medicine Sciences, Zhengzhou 450000, P. R. China;
4. Henan Province Artificial Intelligence Engineering Research Center for Bone Injury Rehabilitation, Zhengzhou 450000, P. R. China;
5. Henan University of Chinese Medicine, Zhengzhou 450000, P. R. China;

Corresponding author：

CAO Xiangyang, Email: Cxy1260@126.com

Keywords：

[Key worlds] Sarcopenia; Artificial intelligence; Large language model; Clinicians; Diagnosis and treatment applications

DOI：

10.7507/1672-2531.202411134

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To evaluate the quality differences in recommendations generated by large language models (LLMs) and clinical practitioners for sarcopenia-related questions. Methods A sarcopenia knowledge base was constructed based on the latest domestic and international research and consensus guidelines. Using the Python environment, a locally deployed and sarcopenia-focused hybrid vertical LLM (referred to as LC) was implemented via LangChain-LLM. Eight fixed questions covering etiology, diagnosis, and prevention were selected, along with eight virtual patient cases. The evaluation team assessed the quality of answers generated by LC and written by clinical practitioners. Quantitative analysis was performed on the precision, recall, and F1 scores (harmonic mean of precision and recall) of treatment recommendations. Results The responses were generally perceived as "possibly written by humans or AI", with a stronger inclination toward being AI-generated, although the accuracy of such judgments was low. Regarding answer quality attributes, LC's responses were superior to those of clinical practitioners in guideline consistency (P<0.01), exhibited similar acceptability (P>0.05), showed better practicality (P<0.05), and had a lower proportion of "1–2 errors" (P<0.05). Quantitative analysis of treatment recommendations indicated that LC and GPT-4.0 outperformed clinical practitioners in recall and F1 scores (P<0.05), with minimal differences between LC and GPT-4.0. Conclusion The locally deployed sarcopenia-focused hybrid vertical LLM demonstrates high accuracy and applicability in addressing sarcopenia-related issues, outperforming clinical practitioners and exhibiting strong clinical decision-support capabilities.

1.	Cruz-Jentoft AJ, Sayer AA. Sarcopenia. Lancet, 2019, 393(10191): 2636-2646.
2.	Polyzos SA, Vachliotis ID, Mantzoros CS. Sarcopenia, sarcopenic obesity and nonalcoholic fatty liver disease. Metabolism, 2023 Oct: 147: 155676.
3.	No authors listed. Will ChatGPT transform healthcare. Nat Med, 2023, 29(3): 505-506.
4.	Moon JH, Lee H, Shin W, et al. Multi-Modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training. IEEE J Biomed Health Inform, 2022, 26(12): 6070-6080.
5.	荣国伟, 孙宝, 王文婧, 等. 大语言模型在数学建模领域的应用探索. 科技创新与应用, 2024, 14(22): 18-21.
6.	Yaneva V, Baldwin P, Jurich DP, et al. Examining ChatGPT performance on USMLE sample items and implications for assessment. Acad Med, 2024, 99(2): 192-197.
7.	克鲁兹·詹托夫特美. 胡亦新, 译. 肌少症: 原书第2版. 北京: 中国科学技术出版社, 2023.
8.	孙建琴. 肌不可失: 肌少症防治全攻略. 上海: 上海科学普及出版社, 2022.
9.	闫炳霖. 骨骼肌减少症. 北京: 科学技术文献出版社, 2021.
10.	Benary M, Wang XD, Schmidt M, et al. Leveraging large language models for decision support in personalized oncology. JAMA Netw Open, 2023, 6(11): e2343689.
11.	邢丹, 王斌, 侯云飞, 等. 中国骨科临床实践指南的质量评价. 中国循证医学杂志, 2017, 17(3): 347-356.
12.	Bernstein IA, Zhang YV, Govil D, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. JAMA Netw Open, 2023, 6(8): e2330320.
13.	夏宇隆, 蒋理, 但炜, 等. 基于人工智能的高血压性脑出血医疗文本信息自动识别系统. 重庆医科大学学报, 2023, 48(9): 1122-1127.
14.	张芊, 陈攀峰, 冯林坤, 等. PeMeBench: 中文儿科医疗问答基准测试方法. 大数据, 2024, 10(5): 28-44.
15.	陈龙飞, 高鑫, 侯皓天, 等. 生成式大语言模型在中文放射医学领域的应用研究. 计算机科学与探索, 2024, 18(9): 2337-2348.
16.	Zhang C, Liu S, Zhou X, et al. Examining the role of large language models in orthopedics: systematic review. J Med Internet Res, 2024, 26: e59607.
17.	Zhang F, Sun J. Multi-feature intelligent oral english error correction based on few-shot learning technology. Comput Intell Neurosci, 2022, 2022: 2501693.
18.	Gallifant J, Fiske A, Levites Strekalova YA, et al. Peer review of GPT-4 technical report and systems card. PLOS Digit Health, 2024, 3(1): e0000417.
19.	Taylor N, Ghose U, Rohanian O, et al. Efficiency at scale: investigating the performance of diminutive language models in clinical tasks. Artif Intell Med, 2024, 157: 103002.
20.	Li J, Deng Y, Sun Q, et al. Benchmarking large language models in evidence-based medicine. IEEE J Biomed Health Inform, 2024, 21: PP.
21.	Kanjee Z, Crowe B, Rodman A. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA, 2023, 330(1): 78-80.
22.	Su Z, Jin K, Wu H, et al. Assessment of large language models in cataract care information provision: a quantitative comparison. Ophthalmol Ther, 2025, 14(1): 103-116.
23.	Fong A, Adams KT, Boxley C, et al. Does one size fit all. Developing an evaluation strategy to assess large language models for patient safety event report analysis. JAMIA Open, 2024, 7(4): ooae128.
24.	Wilhelm TI, Roos J, Kaczmarczyk R. Large language models for therapy recommendations across 3 clinical specialties: comparative study. J Med Internet Res, 2023, 25: e49324.
25.	Teckwani SH, Wong AH, Luke NV, et al. Accuracy and reliability of large language models in assessing learning outcomes achievement across cognitive domains. Adv Physiol Educ, 2024, 48(4): 904-914.

1. Cruz-Jentoft AJ, Sayer AA. Sarcopenia. Lancet, 2019, 393(10191): 2636-2646.
2. Polyzos SA, Vachliotis ID, Mantzoros CS. Sarcopenia, sarcopenic obesity and nonalcoholic fatty liver disease. Metabolism, 2023 Oct: 147: 155676.
3. No authors listed. Will ChatGPT transform healthcare. Nat Med, 2023, 29(3): 505-506.
4. Moon JH, Lee H, Shin W, et al. Multi-Modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training. IEEE J Biomed Health Inform, 2022, 26(12): 6070-6080.
5. 荣国伟, 孙宝, 王文婧, 等. 大语言模型在数学建模领域的应用探索. 科技创新与应用, 2024, 14(22): 18-21.
6. Yaneva V, Baldwin P, Jurich DP, et al. Examining ChatGPT performance on USMLE sample items and implications for assessment. Acad Med, 2024, 99(2): 192-197.
7. 克鲁兹·詹托夫特美. 胡亦新, 译. 肌少症: 原书第2版. 北京: 中国科学技术出版社, 2023.
8. 孙建琴. 肌不可失: 肌少症防治全攻略. 上海: 上海科学普及出版社, 2022.
9. 闫炳霖. 骨骼肌减少症. 北京: 科学技术文献出版社, 2021.
10. Benary M, Wang XD, Schmidt M, et al. Leveraging large language models for decision support in personalized oncology. JAMA Netw Open, 2023, 6(11): e2343689.
11. 邢丹, 王斌, 侯云飞, 等. 中国骨科临床实践指南的质量评价. 中国循证医学杂志, 2017, 17(3): 347-356.
12. Bernstein IA, Zhang YV, Govil D, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. JAMA Netw Open, 2023, 6(8): e2330320.
13. 夏宇隆, 蒋理, 但炜, 等. 基于人工智能的高血压性脑出血医疗文本信息自动识别系统. 重庆医科大学学报, 2023, 48(9): 1122-1127.
14. 张芊, 陈攀峰, 冯林坤, 等. PeMeBench: 中文儿科医疗问答基准测试方法. 大数据, 2024, 10(5): 28-44.
15. 陈龙飞, 高鑫, 侯皓天, 等. 生成式大语言模型在中文放射医学领域的应用研究. 计算机科学与探索, 2024, 18(9): 2337-2348.
16. Zhang C, Liu S, Zhou X, et al. Examining the role of large language models in orthopedics: systematic review. J Med Internet Res, 2024, 26: e59607.
17. Zhang F, Sun J. Multi-feature intelligent oral english error correction based on few-shot learning technology. Comput Intell Neurosci, 2022, 2022: 2501693.
18. Gallifant J, Fiske A, Levites Strekalova YA, et al. Peer review of GPT-4 technical report and systems card. PLOS Digit Health, 2024, 3(1): e0000417.
19. Taylor N, Ghose U, Rohanian O, et al. Efficiency at scale: investigating the performance of diminutive language models in clinical tasks. Artif Intell Med, 2024, 157: 103002.
20. Li J, Deng Y, Sun Q, et al. Benchmarking large language models in evidence-based medicine. IEEE J Biomed Health Inform, 2024, 21: PP.
21. Kanjee Z, Crowe B, Rodman A. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA, 2023, 330(1): 78-80.
22. Su Z, Jin K, Wu H, et al. Assessment of large language models in cataract care information provision: a quantitative comparison. Ophthalmol Ther, 2025, 14(1): 103-116.
23. Fong A, Adams KT, Boxley C, et al. Does one size fit all. Developing an evaluation strategy to assess large language models for patient safety event report analysis. JAMIA Open, 2024, 7(4): ooae128.
24. Wilhelm TI, Roos J, Kaczmarczyk R. Large language models for therapy recommendations across 3 clinical specialties: comparative study. J Med Internet Res, 2023, 25: e49324.
25. Teckwani SH, Wong AH, Luke NV, et al. Accuracy and reliability of large language models in assessing learning outcomes achievement across cognitive domains. Adv Physiol Educ, 2024, 48(4): 904-914.

Chinese Journal of Evidence-Based Medicine

Latest ArticlesApplication of large language models in sarcopenia diagnosis and treatment: a comparative study with clinical decision-making by physicians

Abstract Full text Figures/Tables Video References Cited by

Format

Content