Reply to: Beyond Discrimination: A Call for Comprehensive Assessment of Clinical Prediction Models in Inflammatory Bowel Disease

To The Editors,

We thank Alsoud and Van Calster for their thoughtful and detailed comments on our study, “Validation of ASCVD Risk Prediction Models in Patients With Inflammatory Bowel Disease Using UK Biobank Data,” by Alayo et al.

We appreciate the opportunity to address the concerns you have raised, particularly regarding the methodology of our statistical analysis and the interpretation of our results.

Firstly, we acknowledge the importance of evaluating both discrimination and calibration when assessing the performance of risk prediction models.

Our goal was to investigate the performance of the European Society of Cardiology Systematic Coronary Risk Evaluation 2 (SCORE2) and the 2018 American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE)—2 commonly used ASCVD risk prediction models among IBD patients.

The findings of our study, as you noted, indicate that the model does not generalize well to the IBD population. This result is significant in itself, as it highlights a gap in the model’s generalizability and opens avenues for future research. It underscores the need for models that can accurately predict outcomes for IBD patients, possibly through methodologies that involve retraining these models with IBD-specific data, followed by calibration to refine probability estimates for this cohort. Such an endeavor would require having access to the original model inherent configurations, training data, as well as model hyperparameters, which we do not have. We acknowledge this as a limitation of our study.

Regarding bootstrapping, we opted for stratified k-fold cross-validation because it aligns closely with the specific aims and context of our research. These considerations include reliability for data imbalance, efficiency of handling large data, and tolerance for high-variance models. Bootstrapping on the other hand, could lead to an overestimation of model performance due to the high likelihood of resampling similar instances multiple times, is computationally expensive, and may overestimate area under the ROC curve (AUC) for high-variance models such as this.^1,2

Finally, censoring was not an issue in our study, as all participants had more than 10 years of follow-up and the models estimated the 10-year risk of ASCVD.

Funding

This work was supported by Junior Faculty Development Award from the American College of Gastroenterology to P.D. and IBD Plexus of the Crohn’s & Colitis Foundation to P.D.

Conflicts of Interest

Q.A., D.F.: Nothing to disclose.

P.D.: Research support under a sponsored research agreement unrelated to the data in the article and/or consulting from AbbVie, Arena Pharmaceuticals, Boehringer Ingelheim, Bristol Myers Squibb, Janssen, Pfizer, Prometheus Biosciences, Takeda Pharmaceuticals, Scipher Medicine, and CorEvitas, LLC.

References