Letter regarding “Association of liver abnormalities with in-hospital mortality in patients with COVID-19”

With great enthusiasm, we read a breakthrough study demonstrating the association of liver abnormalities with in-hospital mortality in patients with COVID-19. Integrating 10 candidate predictor parameters determined by multivariate regression analyses, Ding and colleagues formulated a novel prognostic nomogram to predict the survival of patients with COVID-19. However, several crucial pitfalls should be taken into account.

In the study of Ding et al., a total of 2,073 patients with COVID-19 were involved. Subsequently, 10 candidate predictor parameters (including age, severe pneumonia, lymphocyte count, PLT, CRP, D-dimer, creatinine, cTnl, AST, and D-Bil) were selected to construct the prognostic model and plot the nomogram. However, is the limited sample size large enough to establish the prediction model which can predict the overall survival probability of patients with COVID-19? As a matter of fact, we are skeptical about the reliability of this nomogram. In the authoritative methodological research published in the journal of BMJ, Riley et al. pointed out that a robust prediction model should be constructed based on the required sample size that is large enough to sufficiently target precious model prediction and minimize model overfitting. Thus, the required sample size of Ding et al. was calculated according to formulas of Riley et al.^, Firstly, lnL_null is $-\mathrm{5}.\mathrm{605}\left(\mathrm{0}.\mathrm{010}\times \mathrm{100}\times ln\left(\mathrm{0}.\mathrm{010}\right)-\mathrm{0}.\mathrm{010}\times \mathrm{100}\right)$, and max(R²_CS) is $0.106\left(1-\exp \left(\frac{2\times \left(-5.605\right)}{100}\right)\right)$. Secondly, the conservative value of R²_cs is 0.106$\times$8%=0.008. Thirdly, we inputed a key time point to predict the overall survival probability (14 days = 0.038 year), alongside the number of candidate predictor parameters (n = 10), the anticipated mean follow-up (37.81 days = 0.104 year), the mortality rate (0.096), and the conservative value of R²_cs (0.008). Finally, the minimum required sample size of 14-day survival prediction for Ding et al.’s study was calculated in Stata software with the following codes:

pmsampsize, type(s) rsquared(0.008) parameters(10) rate(0.096) timepoint(0.038) meanfup(0.104)

The result is shown in Fig. 1A, indicating that at least 11,200 patients are required for 14-day survival prediction, corresponding to 1,164.8 deaths and an EPP (events per candidate predictor parameter) of 11.18. In addition, the minimum required sample sizes of 21-day and 28-day survival prediction were also calculated (Fig. 1B and C). Therefore, the minimum required sample size to construct the prognostic nomogram should be 11,200 patients, which is much larger than the sample size (2,073 patients) of Ding et al.’s study.

Moreover, though the calibration curve showed good consistency, the ROC (receiver operating characteristic) curve and DCA (decision curve analysis) curves were eagerly warranted.^, The ROC curve can evaluate the discriminative ability of the nomogram, while the DCA curve can evaluate the clinical usefulness of the nomogram. Furthermore, Ding et al.’s nomogram would be more valuable if they applied the disease-specific survival to establish the prognostic nomogram. After all, the overall survival may be impacted by other factors except for COVID-19.

We appreciate Ding et al. for their vital thoughts on the association between liver abnormalities and COVID-19, which has paved a novel way to predict the overall survival probability of patients with COVID-19. However, the minimum required sample size to construct the prognostic nomogram in Ding et al.’s study should be 11,200 patients. The discriminative ability and the clinical usefulness of the nomogram were definitely necessary. Consequently, these above issues may strongly impact the reliability and applicability of Ding et al.’s nomogram.