Skip to main content

Research Repository

Advanced Search

GRADE concept paper 2: Concepts for Judging Certainty on the Calibration of Prognostic Models in a Body of Validation Studies.

GRADE concept paper 2: Concepts for Judging Certainty on the Calibration of Prognostic Models in a Body of Validation Studies. Thumbnail


Abstract

Prognostic models combine several prognostic factors to provide an estimate of the likelihood (or risk) of future events in individual patients, conditional on their prognostic factor values. A fundamental part of evaluating prognostic models is undertaking studies to determine whether their predictive performance, such as calibration and discrimination, is reproduced across settings. Systematic reviews and meta-analyses of studies evaluating prognostic models' performance are a necessary step for selection of models for clinical practice and for testing the underlying assumption that their use will improve outcomes, including patient's reassurance and optimal future planning. In this paper, we highlight key concepts in evaluating the certainty of evidence regarding the calibration of prognostic models. Four concepts are key to evaluating the certainty of evidence on prognostic models' performance regarding calibration. The first concept is that the inference regarding calibration may take one of two forms: deciding whether one is rating certainty that a model's performance is satisfactory or, instead, unsatisfactory, in either case defining the threshold for satisfactory (or unsatisfactory) model performance. Second, inconsistency is the critical GRADE domain to deciding whether we are rating certainty in the model performance being satisfactory or unsatisfactory. Third, depending on whether one is rating certainty in satisfactory or unsatisfactory performance, different patterns of inconsistency of results across studies will inform ratings of certainty of evidence. Fourth, exploring the distribution of point estimates of observed to expected ratio across individual studies, and its determinants, will bear on the need for and direction of future research.

Citation

(2021). GRADE concept paper 2: Concepts for Judging Certainty on the Calibration of Prognostic Models in a Body of Validation Studies. Journal of Clinical Epidemiology, 202-211. https://doi.org/10.1016/j.jclinepi.2021.11.024

Acceptance Date Nov 10, 2021
Publication Date Nov 17, 2021
Journal Journal of Clinical Epidemiology
Print ISSN 0895-4356
Publisher Elsevier
Pages 202-211
DOI https://doi.org/10.1016/j.jclinepi.2021.11.024
Keywords GRADE; Certainty in evidence; Prognosis; Prognostic models; Systematic review; Discrimination; Calibration; Meta-Analysis
Publisher URL https://www.jclinepi.com/article/S0895-4356(21)00376-0/fulltext

Files




Downloadable Citations