Model performance validated under temporal data shift. (A) Validation of discrimination. (B) Validation of calibration. All models were trained on patients who had baseline before 2000 and validated on patients who had baseline between 2000 and 2010 with 1-year interval. The internally validated AUROC (1999-2000) is provided for comparison in (A); and the diagonal line in (B) is the reference calibration curve. Colours in (A) represent the models for validation and the colours in (B) represent the year of baseline for validation. (A) Shows all models suffer performance decline under temporal data shifts. This is represented as AUROC values decrease as the gap between the reference (i.e. 1999–2000) and the baseline of the validated cohort increases. Similarly, as the gap increases, the notable deviations of the calibration curve under temporal shifts for some models in (B) show they are more prone to the detrimental effects of data shifts.