Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model.

Ensor, Joie; Snell, Kym I. E.; Debray, Thomas P. A.; Lambert, Paul C.; Look, Maxime P.; Mamas, Mamas A.; Moons, Karel G. M.; Riley, Richard D.

Abstract

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.

Citation

Ensor, J., Snell, K. I. E., Debray, T. P. A., Lambert, P. C., Look, M. P., Mamas, M. A., …Riley, R. D. (2021). Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model. Statistics in Medicine, 40(13), 3066-3084. https://doi.org/10.1002/sim.8959

• Journal Article Type: Article
• Acceptance Date: Mar 5, 2021
• Online Publication Date: Mar 26, 2021
• Publication Date: Jun 15, 2021
• Journal: Statistics in Medicine
• Print ISSN: 0277-6715
• Publisher: Wiley
• Volume: 40
• Issue: 13
• Pages: 3066-3084
• DOI: https://doi.org/10.1002/sim.8959
• Keywords: external validation; IPD meta-analysis; model recalibration; model updating; time-to-event models
• Publisher URL: https://onlinelibrary.wiley.com/doi/10.1002/sim.8959