Advertisement

Expert opinion vs. meta-analysis: To be or not to be?

  • Calogero Cammà
    Correspondence
    Corresponding author. Address: Section of Gastroenterology & Hepatology, Department of Health Promotion, Mother and Child Care, Internal Medicine and Medical Specialties, PROMISE, University of Palermo, Piazza delle Cliniche n.2, 90127, Palermo, Italy.
    Affiliations
    Section of Gastroenterology & Hepatology, Department of Health Promotion, Mother and Child Care, Internal Medicine and Medical Specialties, PROMISE, University of Palermo, Palermo, Italy
    Search for articles by this author
  • Marco Enea
    Affiliations
    Department of Health Promotion, Mother and Child Care, Internal Medicine and Medical Specialties, PROMISE, University of Palermo, Palermo, Italy
    Search for articles by this author
  • Ciro Celsa
    Affiliations
    Section of Gastroenterology & Hepatology, Department of Health Promotion, Mother and Child Care, Internal Medicine and Medical Specialties, PROMISE, University of Palermo, Palermo, Italy

    Department of Surgical, Oncological and Oral Sciences (Di.Chir.On.S.), University of Palermo, Palermo, Italy
    Search for articles by this author
Published:September 29, 2022DOI:https://doi.org/10.1016/j.jhep.2022.09.017

      Linked Article

      To the Editor:
      Papatheodoridis et al. report a meta-analysis
      • Papatheodoridis G.V.
      • Lekakis V.
      • Voulgaris T.
      • Lampertico P.
      • Berg T.
      • Chan H.L.Y.
      • et al.
      Hepatitis B virus reactivation associated with new classes of immunosuppressants and immunomodulators: a systematic review, meta-analysis, and expert opinion.
      of aggregate data from non-randomized (prospective or retrospective) studies evaluating the risk of HBV reactivation (HBVr) associated with new classes of immunomodulators and immunosuppressants. We fully agree with the study recommendations provided by the expert panel, but some concerns remain about the methodological approach used for the quantitative assessment of the risk of HBVr in this meta-analysis.
      • 1)
        We fully agree with authors that the strength of evidence is considered high when no significant statistical heterogeneity is found among pooled studies. However, it is well known that uncertainty remains in heterogeneity estimates, and this estimate can have large uncertainty which must be considered when interpreting the strength of evidence. The authors evaluated statistical heterogeneity by I2 and τ2 and they stated that there was no significant heterogeneity in almost all pooled risk estimates. All statistical tests for heterogeneity are weak, including I2.
        • Ioannidis J.P.
        • Patsopoulos N.A.
        • Evangelou E.
        Uncertainty in heterogeneity estimates in meta-analyses.
        The clinical implications of this are considerable and must be examined on a case-by-case basis. Putting too much trust in homogeneity of effects may give a false sense of reassurance that one size fits all. Lack of evidence of heterogeneity is not evidence of homogeneity. In four pooled analyses (Figs 1C, 1D, 2D and 2E of the paper) I2 was 0%. Meta-analyses where I2 is estimated at 0% are affected by an important misconception. Many authors interpret this as an absence of heterogeneity, but the upper 95% CI limit may be substantiated. Confidence intervals of I2 should be calculated and considered when interpreting meta-analyses. Could you please provide the 95% CI of I2?
      • 2)
        We fully agree with the authors that prediction intervals should be routinely reported to allow more informative inference in meta-analysis. However, the estimate of the prediction interval will be large if the estimates of the summary effect and between-study heterogeneity are imprecise. For example, if they are based on only a few small studies.
        • IntHout J.
        • Ioannidis J.P.
        • Rovers M.M.
        • Goeman J.J.
        Plea for routinely presenting prediction intervals in meta-analysis.
        In the very clinically relevant setting of immune checkpoint inhibitors in HBsAg+ patients without nucleoside analogue prophylaxis (Fig. 2B of the paper), prediction intervals range from 0 to 0.99. Is this quantitative estimate useful to support the recommendation?
      • 3)
        HBVr is an uncommon event during treatment with some classes of immunosuppressive drugs, especially in patients receiving nucleoside analogue prophylaxis, as proven in a large amount of included studies, that reported 0 events. The authors reported that generalized linear mixed model was used, but it is not clear what method was used to handle rare events.
        • Zhou Q.
        • Li Q.
        • Meng W.
        • Luo Z.
        • Chen Y.
        Statistical concerns for meta-analysis of rare events and small sample sizes.
        Please verify if the application of proper methods for meta-analyses of rare events (for example, beta-binomial model, which seems to be more suitable for rare events than binomial normal models
        • Ma Y.
        • Chu H.
        • Mazumdar M.
        Meta-analysis of proportions of rare events-A comparison of exact likelihood methods with robust variance estimation.
        – the method that we assumed the authors applied) provides similar estimates.
      • 4)
        The authors did not perform meta-regression analysis. We suggest exploring if study-level covariates (sample size, single-vs. multicentre, study design, and finally study quality) may be a potential source of heterogeneity. Moreover, there are many patient-level characteristics (i.e. ethnicity and the underlying disease needing immunosuppressive treatment) that can affect the risk of HBVr. Unfortunately, the sources of variability of the included studies were not reported with enough consistency, particularly regarding patient-level covariates. So, the results obtained by this meta-analysis of aggregate data may easily be affected by ecological bias. More accurate treatment comparisons could be achieved only by a meta-analysis of individual patient data.
      • 5)
        The authors restricted the systematic review and meta-analysis to only studies where a single drug or drug class was used. However, in real-life clinical practice, most patients receive combination regimens including more drugs with different mechanisms of action associated with different risks of HBVr. This issue can limit the generalizability of these results to several clinical practice settings.
      • 6)
        Although the authors stated that the meta-analysis was performed according to PRISMA-P, they did not report a formal quality assessment, nor did they assess the risk of bias of individual studies. Moreover, publication bias across studies should be assessed.
      • 7)
        Non-randomized studies may experience many problems that could reduce their internal and external validity.
        • Ozminkowski R.J.
        • Wortman P.M.
        • Roloff D.W.
        Inborn/outborn status and neonatal survival: a meta-analysis of non-randomized studies.
        When assessing retrospective studies, the most important bias is the likelihood of inappropriate selection of patients for treatment, which can lead to incorrect results and spurious associations. Although the authors were very fair in underlining the limitation related to the non-consecutive selection of patients studied, we suggest exploring if study design (prospective vs. retrospective) may be a significant source of heterogeneity using sensitivity analyses and meta-regressions.

      Financial support

      The authors received no financial support to produce this manuscript.

      Conflict of interest

      Prof. Cammà is on the speaker’s bureau for Eisai, Ipsen, AstraZeneca, and Roche. Dr. Celsa is on the speakers’ bureau for Eisai, MSD, and Ipsen. Marco Enea has no conflict of interest.
      Please refer to the accompanying ICMJE disclosure forms for further details.

      Authors’ contributions

      Calogero Cammà, Marco Enea and Ciro Celsa conceived, reviewed and approved the manuscript.

      Supplementary data

      The following are the supplementary data to this article:

      References

        • Papatheodoridis G.V.
        • Lekakis V.
        • Voulgaris T.
        • Lampertico P.
        • Berg T.
        • Chan H.L.Y.
        • et al.
        Hepatitis B virus reactivation associated with new classes of immunosuppressants and immunomodulators: a systematic review, meta-analysis, and expert opinion.
        J Hepatol. 2022 Jul 16; (S0168-8278(22)02935-X. https://doi.org/10.1016/j.jhep.2022.07.003. [Epub ahead of print])
        • Ioannidis J.P.
        • Patsopoulos N.A.
        • Evangelou E.
        Uncertainty in heterogeneity estimates in meta-analyses.
        BMJ. 2007 Nov 3; 335: 914-916
        • IntHout J.
        • Ioannidis J.P.
        • Rovers M.M.
        • Goeman J.J.
        Plea for routinely presenting prediction intervals in meta-analysis.
        BMJ Open. 2016 Jul 12; 6e010247
        • Zhou Q.
        • Li Q.
        • Meng W.
        • Luo Z.
        • Chen Y.
        Statistical concerns for meta-analysis of rare events and small sample sizes.
        Lancet Infect Dis. 2022 Aug; 22: 1111
        • Ma Y.
        • Chu H.
        • Mazumdar M.
        Meta-analysis of proportions of rare events-A comparison of exact likelihood methods with robust variance estimation.
        Commun Stat Simul Comput. 2016; 45: 3036-3052
        • Ozminkowski R.J.
        • Wortman P.M.
        • Roloff D.W.
        Inborn/outborn status and neonatal survival: a meta-analysis of non-randomized studies.
        Stat Med. 1988 Dec; 7: 1207-1221