Fragility index for assessing the robustness of randomized controlled trials_Chinese Journal of Evidence-Based Medicine

Authors：

LUO Minjing ^1,2,3,4 , LIU Zhihan ^1,2,3,4 , HUANG Jinghan ⁴ , WANG Yingqiao ⁴ , LI Yilin ⁴ , LIU Meijun ^1,2,3,4 , TAO Yunci ^1,2,3,4 , CAO Rui ^1,2,3,4 , WANG Yaqi ^1,2,3,4 , LIU Jianping ^1,2,3,4 , ZHANG Ying ^1,2,3,4 ,  FEI Yutong ^1,2,3,4

1. Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing 100029, P. R. China;
2. International Institute of Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing 100029, P. R. China;
3. Beijing GRADE Center, Beijing 100029, P. R. China;
4. Beijing University of Chinese Medicine, Beijing 100029, P. R. China;

Corresponding author：

FEI Yutong, Email: feiyt@bucm.edu.cn

Keywords：

Randomized controlled trial; Fragility index; Statistical significance; Clinical significance; Clinical decision

DOI：

10.7507/1672-2531.202307022

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

After the completion of a clinical trial, its conclusion generally depends on the results of statistical analysis of the main outcome, that is, whether the P-value in the hypothesis test is less than the α level of the hypothesis test, usually α=0.05. The size of the P-value indicates the sufficient degree of reason for making the hypothesis judgment, and can be interpreted as to determine whether a conclusion is statistically significant but does not involve the difference in the degree of drug effects or other effects. Fragility index, which is, the minimum number of patients required to change the occurrence of a target outcome event to a non-target outcome event from a statistically significant outcome to a non-significant outcome, can be used to assist in understanding of clinical trial statistical inference results and assisting in clinical decision making This paper discusses the concept, calculation method and clinical application of the fragility index, and recommends that the fragility index be routinely reported in all future randomized controlled trials to help patient clinicians and policymakers make appropriate and optimal decisions.

Citation： LUO Minjing, LIU Zhihan, HUANG Jinghan, WANG Yingqiao, LI Yilin, LIU Meijun, TAO Yunci, CAO Rui, WANG Yaqi, LIU Jianping, ZHANG Ying, FEI Yutong. Fragility index for assessing the robustness of randomized controlled trials. Chinese Journal of Evidence-Based Medicine, 2024, 24(2): 243-248. doi: 10.7507/1672-2531.202307022 Copy

1.	Fisher R. Statistical methods for research workers (13th edition). New York: Hafner, 1958.
2.	姚晨. 医学研究结论的统计学推断. 北京大学学报(医学版), 2007, 39(2): 213.
3.	Marks HM. Rigorous uncertainty: why RA Fisher is important. Int J Epidemiol, 2003, 32(6): 932-937.
4.	Senn S. Significant errors. Lancet, 2010, 376(9750): 1390-1391.
5.	Ziliak ST. The Validus Medicus and a new gold standard. Lancet, 2010, 376(9738): 324-325.
6.	Nuzzo R. Scientific method: statistical errors. Nature, 2014, 506(7487): 150-152.
7.	Nagarajan PJ, Garla BK, Taranath M, et al. The file drawer effect: a call for meticulous methodology and tolerance for non-significant results. Indian J Anaesth, 2017, 61(6): 516-517.
8.	周敏林. 临床试验中P值的意义及结果解读. 肾脏病与透析肾移植杂志, 2017, 26(2): 170-173,183.
9.	Chavalarias D, Wallach JD, Li AH, et al. Evolution of reporting P values in the biomedical literature, 1990-2015. JAMA, 2016, 315(11): 1141-1148.
10.	Kyriacou DN. The enduring evolution of the P value. JAMA, 2016, 315(11): 1113-1115.
11.	Wasserstein RL, Lazar NA. The ASA’s statement on P-value: context, proces, and purpose. Am Stat, 2016, 70(2): 129-133.
12.	Feinstein AR. The unit fragility index: an additional appraisal of "statistical significance" for a contrast of two proportions. J Clin Epidemiol, 1990, 43(2): 201-209.
13.	Walsh M, Srinathan SK, McAuley DF, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility index. J Clin Epidemiol, 2014, 67(6): 622-628.
14.	Carlbring P, Hägglund M, Luthström A, et al. Internet-based behavioral activation and acceptance-based treatment for depression: a randomized controlled trial. J Affect Disord, 2013, 148(2-3): 331-337.
15.	Ho AK. The fragility index for assessing the robustness of the statistically significant results of experimental clinical studies. J Gen Intern Med, 2022, 37(1): 206-211.
16.	Dettori JR, Norvell DC. How fragile are the results of a trial. The fragility index. Global Spine J, 2020, 10(7): 940-942.
17.	Nguyen TL, Landais P. Randomized controlled trials: significant results-fragile, though. Kidney Int, 2017, 92(6): 1319-1320.
18.	陶刚, 郑小卫, 张轶雯, 等. 随机对照试验设计及实践心得. 中国现代应用药学, 2022, 39(24): 3280-3283.
19.	Evaniew N, Files C, Smith C, et al. The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey. Spine J, 2015, 15(10): 2188-2197.
20.	Ridgeon EE, Young PJ, Bellomo R, et al. The fragility index in multicenter randomized controlled critical care trials. Crit Care Med, 2016, 44(7): 1278-1284.
21.	Khan M, Evaniew N, Gichuru M, et al. The fragility of statistically significant findings from randomized trials in sports surgery: a systematic survey. Am J Sports Med, 2017, 45(9): 2164-2170.
22.	Shochet LR, Kerr PG, Polkinghorne KR. The fragility of significant results underscores the need of larger randomized controlled trials in nephrology. Kidney Int, 2017, 92(6): 1469-1475.
23.	Shen C, Shamsudeen I, Farrokhyar F, et al. Fragility of results in ophthalmology randomized controlled trials: a systematic review. Ophthalmology, 2018, 125(5): 642-648.
24.	Noel CW, McMullen C, Yao C, et al. The fragility of statistically significant findings from randomized trials in head and neck surgery. Laryngoscope, 2018, 128(9): 2094-2100.
25.	Edwards E, Wayant C, Besas J, et al. How fragile are clinical trial outcomes that support the CHEST clinical practice guidelines for VTE. Chest, 2018, 154(3): 512-520.
26.	Narayan VM, Gandhi S, Chrouser K, et al. The fragility of statistically significant findings from randomised controlled trials in the urological literature. BJU Int, 2018, 122(1): 160-166.
27.	Grolleau F, Collins GS, Smarandache A, et al. The fragility and reliability of conclusions of anesthesia and critical care randomized trials with statistically significant findings: a systematic review. Crit Care Med, 2019, 47(3): 456-462.
28.	Bertaggia L, Baiardo Redaelli M, Lembo R, et al. The fragility index in peri-operative randomised trials that reported significant mortality effects in adults. Anaesthesia, 2019, 74(8): 1057-1060.
29.	Khan MS, Ochani RK, Shaikh A, et al. Fragility index in cardiovascular randomized controlled trials. Circ Cardiovasc Qual Outcomes, 2019, 12(12): e005755.
30.	Brown J, Lane A, Cooper C, et al. The results of randomized controlled trials in emergency medicine are frequently fragile. Ann Emerg Med, 2019, 73(6): 565-576.
31.	Del Paggio JC, Tannock IF. The fragility of phase 3 trials supporting FDA-approved anticancer medicines: a retrospective analysis. Lancet Oncol, 2019, 20(8): 1065-1069.
32.	Majeed M, Agrawal R, Attar BM, et al. Fragility index: how fragile is the data that support the American College of Gastroenterology guidelines for the management of Crohn's disease. Eur J Gastroenterol Hepatol, 2020, 32(2): 193-198.
33.	Desnoyers A, Wilson BE, Nadler MB, et al. Fragility index of trials supporting approval of anti-cancer drugs in common solid tumours. Cancer Treat Rev, 2021, 94: 102167.
34.	Maldonado DR, Go CC, Huang BH, et al. The fragility index of hip arthroscopy randomized controlled trials: a systematic survey. Arthroscopy, 2021, 37(6): 1983-1989.
35.	Ioannidis JP. Why most published research findings are false. PLoS Med, 2005, 2(8): e124.
36.	Ioannidis JP, Cappelleri JC, Sacks HS, et al. The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Control Clin Trials, 1997, 18(5): 431-444.
37.	Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA, 2005, 294(2): 218-228.
38.	Ziegler EJ, Fisher CJ, Sprung CL, et al. Treatment of gram-negative bacteremia and septic shock with HA-1A human monoclonal antibody against endotoxin. A randomized, double-blind, placebo-controlled trial. The HA-1A Sepsis Study Group. N Engl J Med, 1991, 324(7): 429-436.
39.	McCloskey RV, Straube RC, Sanders C, et al. Treatment of septic shock with human monoclonal antibody HA-1A. A randomized, double-blind, placebo-controlled trial. CHESS Trial Study Group. Ann Intern Med, 1994, 121(1): 1-5.
40.	Devereaux PJ, Yusuf S. The evolution of the randomized controlled trial and its role in evidence-based decision making. J Intern Med, 2003, 254(2): 105-113.
41.	Tepel M, van der Giet M, Schwarzfeld C, et al. Prevention of radiographic-contrast-agent-induced reductions in renal function by acetylcysteine. N Engl J Med, 2000, 343(3): 180-184.
42.	Carbonell N, Sanjuán R, Blasco M, et al. N-acetylcysteine: short-term clinical benefits after coronary angiography in high-risk renal patients. Rev Esp Cardiol, 2010, 63(1): 12-19.
43.	Rashid ST, Salman M, Myint F, et al. Prevention of contrast-induced nephropathy in vascular patients undergoing angiography: a randomized controlled trial of intravenous N-acetylcysteine. J Vasc Surg, 2004, 40(6): 1136-1141.
44.	Webb JG, Pate GE, Humphries KH, et al. A randomized controlled trial of intravenous N-acetylcysteine for the prevention of contrast-induced nephropathy after cardiac catheterization: lack of effect. Am Heart J, 2004, 148(3): 422-429.
45.	Jaffery Z, Verma A, White CJ, et al. A randomized trial of intravenous n-acetylcysteine to prevent contrast induced nephropathy in acute coronary syndromes. Catheter Cardiovasc Interv, 2012, 79(6): 921-926.
46.	Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol, 2011, 64(12): 1283-1293.
47.	刘金来, 周汉建, 赵长林. 临床治疗性研究的循证医学评价. 循证医学, 2002, 2(3): 177-183.
48.	Chaitoff A, Zheutlin A, Niforatos JD. The fragility index and trial significance. JAMA Intern Med, 2020, 180(11): 1554.

1. Fisher R. Statistical methods for research workers (13th edition). New York: Hafner, 1958.
2. 姚晨. 医学研究结论的统计学推断. 北京大学学报(医学版), 2007, 39(2): 213.
3. Marks HM. Rigorous uncertainty: why RA Fisher is important. Int J Epidemiol, 2003, 32(6): 932-937.
4. Senn S. Significant errors. Lancet, 2010, 376(9750): 1390-1391.
5. Ziliak ST. The Validus Medicus and a new gold standard. Lancet, 2010, 376(9738): 324-325.
6. Nuzzo R. Scientific method: statistical errors. Nature, 2014, 506(7487): 150-152.
7. Nagarajan PJ, Garla BK, Taranath M, et al. The file drawer effect: a call for meticulous methodology and tolerance for non-significant results. Indian J Anaesth, 2017, 61(6): 516-517.
8. 周敏林. 临床试验中P值的意义及结果解读. 肾脏病与透析肾移植杂志, 2017, 26(2): 170-173,183.
9. Chavalarias D, Wallach JD, Li AH, et al. Evolution of reporting P values in the biomedical literature, 1990-2015. JAMA, 2016, 315(11): 1141-1148.
10. Kyriacou DN. The enduring evolution of the P value. JAMA, 2016, 315(11): 1113-1115.
11. Wasserstein RL, Lazar NA. The ASA’s statement on P-value: context, proces, and purpose. Am Stat, 2016, 70(2): 129-133.
12. Feinstein AR. The unit fragility index: an additional appraisal of "statistical significance" for a contrast of two proportions. J Clin Epidemiol, 1990, 43(2): 201-209.
13. Walsh M, Srinathan SK, McAuley DF, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility index. J Clin Epidemiol, 2014, 67(6): 622-628.
14. Carlbring P, Hägglund M, Luthström A, et al. Internet-based behavioral activation and acceptance-based treatment for depression: a randomized controlled trial. J Affect Disord, 2013, 148(2-3): 331-337.
15. Ho AK. The fragility index for assessing the robustness of the statistically significant results of experimental clinical studies. J Gen Intern Med, 2022, 37(1): 206-211.
16. Dettori JR, Norvell DC. How fragile are the results of a trial. The fragility index. Global Spine J, 2020, 10(7): 940-942.
17. Nguyen TL, Landais P. Randomized controlled trials: significant results-fragile, though. Kidney Int, 2017, 92(6): 1319-1320.
18. 陶刚, 郑小卫, 张轶雯, 等. 随机对照试验设计及实践心得. 中国现代应用药学, 2022, 39(24): 3280-3283.
19. Evaniew N, Files C, Smith C, et al. The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey. Spine J, 2015, 15(10): 2188-2197.
20. Ridgeon EE, Young PJ, Bellomo R, et al. The fragility index in multicenter randomized controlled critical care trials. Crit Care Med, 2016, 44(7): 1278-1284.
21. Khan M, Evaniew N, Gichuru M, et al. The fragility of statistically significant findings from randomized trials in sports surgery: a systematic survey. Am J Sports Med, 2017, 45(9): 2164-2170.
22. Shochet LR, Kerr PG, Polkinghorne KR. The fragility of significant results underscores the need of larger randomized controlled trials in nephrology. Kidney Int, 2017, 92(6): 1469-1475.
23. Shen C, Shamsudeen I, Farrokhyar F, et al. Fragility of results in ophthalmology randomized controlled trials: a systematic review. Ophthalmology, 2018, 125(5): 642-648.
24. Noel CW, McMullen C, Yao C, et al. The fragility of statistically significant findings from randomized trials in head and neck surgery. Laryngoscope, 2018, 128(9): 2094-2100.
25. Edwards E, Wayant C, Besas J, et al. How fragile are clinical trial outcomes that support the CHEST clinical practice guidelines for VTE. Chest, 2018, 154(3): 512-520.
26. Narayan VM, Gandhi S, Chrouser K, et al. The fragility of statistically significant findings from randomised controlled trials in the urological literature. BJU Int, 2018, 122(1): 160-166.
27. Grolleau F, Collins GS, Smarandache A, et al. The fragility and reliability of conclusions of anesthesia and critical care randomized trials with statistically significant findings: a systematic review. Crit Care Med, 2019, 47(3): 456-462.
28. Bertaggia L, Baiardo Redaelli M, Lembo R, et al. The fragility index in peri-operative randomised trials that reported significant mortality effects in adults. Anaesthesia, 2019, 74(8): 1057-1060.
29. Khan MS, Ochani RK, Shaikh A, et al. Fragility index in cardiovascular randomized controlled trials. Circ Cardiovasc Qual Outcomes, 2019, 12(12): e005755.
30. Brown J, Lane A, Cooper C, et al. The results of randomized controlled trials in emergency medicine are frequently fragile. Ann Emerg Med, 2019, 73(6): 565-576.
31. Del Paggio JC, Tannock IF. The fragility of phase 3 trials supporting FDA-approved anticancer medicines: a retrospective analysis. Lancet Oncol, 2019, 20(8): 1065-1069.
32. Majeed M, Agrawal R, Attar BM, et al. Fragility index: how fragile is the data that support the American College of Gastroenterology guidelines for the management of Crohn's disease. Eur J Gastroenterol Hepatol, 2020, 32(2): 193-198.
33. Desnoyers A, Wilson BE, Nadler MB, et al. Fragility index of trials supporting approval of anti-cancer drugs in common solid tumours. Cancer Treat Rev, 2021, 94: 102167.
34. Maldonado DR, Go CC, Huang BH, et al. The fragility index of hip arthroscopy randomized controlled trials: a systematic survey. Arthroscopy, 2021, 37(6): 1983-1989.
35. Ioannidis JP. Why most published research findings are false. PLoS Med, 2005, 2(8): e124.
36. Ioannidis JP, Cappelleri JC, Sacks HS, et al. The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Control Clin Trials, 1997, 18(5): 431-444.
37. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA, 2005, 294(2): 218-228.
38. Ziegler EJ, Fisher CJ, Sprung CL, et al. Treatment of gram-negative bacteremia and septic shock with HA-1A human monoclonal antibody against endotoxin. A randomized, double-blind, placebo-controlled trial. The HA-1A Sepsis Study Group. N Engl J Med, 1991, 324(7): 429-436.
39. McCloskey RV, Straube RC, Sanders C, et al. Treatment of septic shock with human monoclonal antibody HA-1A. A randomized, double-blind, placebo-controlled trial. CHESS Trial Study Group. Ann Intern Med, 1994, 121(1): 1-5.
40. Devereaux PJ, Yusuf S. The evolution of the randomized controlled trial and its role in evidence-based decision making. J Intern Med, 2003, 254(2): 105-113.
41. Tepel M, van der Giet M, Schwarzfeld C, et al. Prevention of radiographic-contrast-agent-induced reductions in renal function by acetylcysteine. N Engl J Med, 2000, 343(3): 180-184.
42. Carbonell N, Sanjuán R, Blasco M, et al. N-acetylcysteine: short-term clinical benefits after coronary angiography in high-risk renal patients. Rev Esp Cardiol, 2010, 63(1): 12-19.
43. Rashid ST, Salman M, Myint F, et al. Prevention of contrast-induced nephropathy in vascular patients undergoing angiography: a randomized controlled trial of intravenous N-acetylcysteine. J Vasc Surg, 2004, 40(6): 1136-1141.
44. Webb JG, Pate GE, Humphries KH, et al. A randomized controlled trial of intravenous N-acetylcysteine for the prevention of contrast-induced nephropathy after cardiac catheterization: lack of effect. Am Heart J, 2004, 148(3): 422-429.
45. Jaffery Z, Verma A, White CJ, et al. A randomized trial of intravenous n-acetylcysteine to prevent contrast induced nephropathy in acute coronary syndromes. Catheter Cardiovasc Interv, 2012, 79(6): 921-926.
46. Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol, 2011, 64(12): 1283-1293.
47. 刘金来, 周汉建, 赵长林. 临床治疗性研究的循证医学评价. 循证医学, 2002, 2(3): 177-183.
48. Chaitoff A, Zheutlin A, Niforatos JD. The fragility index and trial significance. JAMA Intern Med, 2020, 180(11): 1554.

Previous Article
Introduction and application of an auxiliary implementation tool (CERBOT) for target trial emulation study

Chinese Journal of Evidence-Based Medicine

Fragility index for assessing the robustness of randomized controlled trials

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Format

Content