Clinical likelihood models calibrated against observed obstructive coronary artery disease on computed tomography angiography

Abstract

Aims

Models predicting the likelihood of obstructive coronary artery disease (CAD) on invasive coronary angiography exist. However, as stable patients with new-onset chest pain frequently have lower clinical likelihood and preferably undergo index testing by non-invasive tests such as coronary computed tomography angiography (CCTA), clinical likelihood models calibrated against observed obstructive CAD at CCTA are warranted. The aim was to develop CCTA-calibrated risk-factor- and coronary artery calcium score-weighted clinical likelihood models (i.e. RF-CL_CCTA and CACS-CL_CCTA models, respectively).

Methods and results

Based on age, sex, symptoms, and cardiovascular risk factors, an advanced machine learning algorithm utilized a training cohort (n = 38 269) of symptomatic outpatients with suspected obstructive CAD to develop both a RF-CL_CCTA model and a CACS-CL_CCTA model to predict observed obstructive CAD on CCTA. The models were validated in several cohorts (n = 28 340) and compared with a currently endorsed basic pre-test probability (Basic PTP) model. For both the training and pooled validation cohorts, observed obstructive CAD at CCTA was defined as >50% diameter stenosis. Observed obstructive CAD at CCTA was present in 6443 (22.7%) patients in the pooled validation cohort. While the Basic PTP underestimated the prevalence of observed obstructive CAD at CCTA, the RF-CL_CCTA and CACS-CL_CCTA models showed superior calibration. Compared with the Basic PTP model, the RF-CL_CCTA and CACS-CL_CCTA models showed superior discrimination (area under the receiver operating curves 0.71 [95% confidence interval (CI) 0.70–0.72] vs. 0.74 (95% CI 0.73–0.75) and 0.87 (95% CI 0.86–0.87), P < 0.001 for both comparisons).

Conclusion

CCTA-calibrated clinical likelihood models improve calibration and discrimination of observed obstructive CAD at CCTA.

Graphical Abstract

Open in new tabDownload slide

Introduction

In patients with new onset, stable symptoms suggestive of obstructive coronary artery disease (CAD), pre-test probability (PTP) estimation is recommended to guide referral for non-invasive testing and treatment decisions.^1,2 Additionally, the post-test probability of obstructive CAD is estimated based on the PTP value combined with likelihood ratios for non-invasive diagnostic tests.³ Thus, PTP model precision affects patient management.

Classically, PTP tables include only sex, age, and chest pain typicality but several studies report contemporary overestimation of such PTP values derived from historical invasive coronary angiography (ICA) and autopsy data sets, and consequently low diagnostic yield of obstructive CAD.^4,5 To improve PTP-guided patient management, the 2019 European Society of Cardiology (ESC) guidelines on chronic coronary syndrome⁶ suggested a recalibrated Basic PTP model⁴ based on contemporary data sets that used a mixed reference standard of ICA and coronary computed tomography angiography (CCTA), which was later endorsed by the recent North-American guidelines.¹ The ESC guideline additionally introduced the novel concept of ‘clinical likelihood’ including risk factors and computed tomography-derived coronary artery calcium score that could be used to modify patient-specific pre-test risk.⁶ However, a clinically feasible and validated tool to estimate this clinical likelihood was also recognized as a ‘gap in evidence’.⁶

To address this gap, simple and clinically useful tools for patient-specific clinical likelihood estimation were proposed.⁷ Against a reference of ICA, the risk factor- and coronary artery calcium score-weighted clinical likelihood models (i.e. the RF-CL and CACS-CL models, respectively) showed improved discrimination of patients with obstructive CAD,^7,8 and both models were recently endorsed by the 2024 ESC guideline to guide the management of stable patients with new-onset chest pain.² However, as (i) lower risk patients account for one in two of de novo chest pain patients,^7,8 (ii) downstream testing of such patients is often guided by results on first-line non-invasive diagnostic testing using e.g. CCTA,² and (iii) the predicted clinical likelihood of obstructive CAD on CCTA is underestimated by models derived against mixed ICA/CCTA data due to underestimation of CAD severity,⁹ clinical likelihood models calibrated solely using a reference of stenosis at CCTA are warranted.

Thus, using a reference standard of observed obstructive CAD at CCTA, we aimed to (i) recalibrate the RF-CL and CACS-CL models (i.e. the RF-CL_CCTA and CACS-CL_CCTA models, respectively), (ii) large-scale externally validate these models, and (iii) propose clinically relevant clinical likelihood categories with concordant CCTA- and ICA-calibrated cut-offs.

Methods

Overview of study design

Based on previous experience outlining the discriminative gain of complementary risk factors and a coronary artery calcium score to a clinical likelihood estimation in patients with new-onset chest pain,⁷ we developed CCTA-calibrated RF-CL_CCTA and CACS-CL_CCTA models to estimate the prevalence of observed obstructive CAD at CCTA in a training cohort of patients without previously documented CAD. Optimal RF-CL_CCTA and CACS-CL_CCTA model thresholds for test referral/deferral were investigated using the observed prevalence of diagnosed obstructive CAD at ICA as reference. Model discrimination and calibration were validated in multiple external validation cohorts of patients without previously documented CAD, similarly utilizing observed obstructive CAD at CCTA as reference standard. For both calibration and discrimination, the RF-CL_CCTA and CACS-CL_CCTA models were compared with a Basic PTP model that did not utilize additional risk factors or CACS to modify the clinical likelihood estimation.^1,4,6

Training cohort

The training cohort included patients who underwent first-time CCTA from 2008 to 2017 in all 13 hospitals in the Western part of Denmark (uptake area 3.3 million; 55% of the total Danish population). All patients were registered in the mandatory regional Danish population-based clinical database, the Western Denmark Heart Registry (WDHR). The WDHR contains information on the clinical evaluation and includes data regarding cardiac risk factors, symptoms at the time of referral for CCTA, and CCTA and ICA results. The training cohort included patients having CACS and interpretable CCTA (n = 38 269).

Validation cohorts

External validation was performed using several cohorts, which combined formed the pooled validation cohort of patients without previously known CAD (n = 28 340):

1. Temporal validation of the RF-CL_CCTA and CACS-CL_CCTA models was performed using the WDHR for patients who underwent first-time CCTA from January 2018 to August 2019 (n = 8844).

2. The Danish study of Non-Invasive testing in Coronary Artery Disease (Dan-NICAD) trial^10–14 where de novo chest pain patients underwent a structured interview by trained study nurses to assess cardiac risk factors and symptoms before clinically indicated CCTA. The validation cohort for this study included patients receiving interpretable CCTA (n = 4369).

3. The Prospective Multi-center Imaging Study for Evaluation of Chest Pain (PROMISE) trial^7,8 where de novo chest pain patients were randomly allocated to a strategy of initial anatomical testing using CCTA or to functional testing. The validation cohort for this study included only those allocated to and receiving interpretable CCTA (n = 4207).

4. The Scottish Computed Tomography of the HEART (SCOT-HEART) trial^15,16 where chest pain patients were randomly allocated to standard care or standard care with additional CCTA. The validation cohort for this study included only those allocated to and receiving interpretable CCTA (n = 1585).

5. A Singapore cohort where de novo chest pain patients underwent clinically indicated CCTA (n = 893).

6. A Finnish cohort where de novo chest pain patients underwent clinically indicated CCTA (n = 2014).

7. An Australian cohort where de novo chest pain patients underwent clinically indicated CCTA (n = 915).

8. A Brazilian cohort where de novo chest pain patients underwent clinically indicated CCTA (n = 4575).

9. A Portuguese cohort where de novo chest pain patients underwent clinically indicated CCTA (n = 1284).

For the present analyses, the Global Pretest probability Study of Coronary Artery Disease (GPS-CAD) (clinicaltrials.gov NCT05722145) shared data for Cohorts 5–7.

Concordant model cut-offs

Using the training cohort (n = 38 269), optimal RF-CL_CCTA and CACS-CL_CCTA model cut-offs for test referral/deferral were investigated using the observed prevalence of diagnosed obstructive CAD at ICA as reference. Obstructive CAD prevalences were grouped according to guideline thresholds [very-low (≤5%), low (>5–≤15%), moderate (>15–≤50%), moderate/high (>50–≤85%), and high (>85%)],² and concordant prevalences of observed obstructive CAD at CCTA were identified.

• Findings were validated in the temporal WDHR validation cohort (n = 8844).

Definition of variables

Basic PTP variables included sex, age and type of chest complaints at admission, the latter categorized by a symptom characteristic score ranging from 0–3 points based (i) discomfort in the chest, neck, jaw, shoulder, or arm of constricting character; (ii) symptoms provoked by exertion or emotional stress; and (iii) symptoms relieved by rest or nitroglycerine. The symptom characteristic score corresponds to previous denotation of typical (3 points), atypical (2 points), and non-anginal (0–1 point) chest pain. Dyspnoea was considered like atypical chest pain (2 points). Thus, atypical chest pain constituted of two-of-three of the criteria covering typical chest pain, and other chest pain symptoms were defined as non-anginal chest pain. Dyspnoea was defined as exertional dyspnoea as the primary symptom and categorized as atypical chest pain in accordance with previous suggestions.⁷

Risk factors included in the RF-CL_CCTA were hypertension, dyslipidaemia, diabetes, smoking, and family history of ischaemic heart disease. Hypertension, dyslipidaemia, and diabetes were defined as either diagnosed by a physician or if receiving medical treatment for these conditions. Smoking was defined as currently smoking or having a history of smoking. The definition of family history constituted of first-degree relatives with early signs of IHD; men < 55 years of age and women < 65 years. The number of risk factor present was grouped (0–1, 2–3, 4–5). CACS was calculated based on non-contrast enhanced CT scans using the Agatston method.

Reference standards of observed obstructive CAD at CCTA and diagnosed obstructive CAD at ICA

In the present study, two different reference standards were utilized:

1. Observed obstructive CAD at CCTA: Across the training and validation cohorts, observed obstructive CAD at CCTA was defined as >50% diameter stenosis by site-reading in a vessel exceeding 2 mm in diameter.

2. Diagnosed obstructive CAD at ICA: For the investigation of concordant RF-CL_CCTA and CACS-CL_CCTA model cut-offs for test referral/deferral according to the observed prevalence of diagnosed obstructive CAD at ICA, diagnosed obstructive CAD at ICA was defined as >50% diameter stenosis within 120 days of the index CCTA.

Statistical analyses

The RF-CL_CCTA and CACS-CL_CCTA models were retrained versions of the RF-CL and CACS-CL regression models.⁷ Model training was done in the same training cohort as the original RF-CL and CACS-CL models but with observed obstructive CAD at CCTA as outcome (see DEVELOPMENT OF THE RF-CL_CCTA and CACS-CL_CCTA MODELS in the Supplementary Material). By model recalibration, model refitting (level 3 recalibration) was performed, thus fully retraining the RF-CL_CCTA and CACS-CL_CCTA against the outcome of observed obstructive CAD at CCTA, but with preservation of model variables and interactions as in the RF-CL and CACS-CL models.

RF-CL_CCTA and CACS-CL_CCTA estimates were stratified by diagnosed obstructive CAD prevalences at ICA. Calibration and discrimination of the Basic PTP, RF-CL_CCTA, and CACS-CL_CCTA models were evaluated according to previous recommendations.¹⁷ The discrimination C-statistic included the area under the receiver operating characteristic (AUC) curve. Categorical net reclassification indexes (NRIs) were calculated for the RF-CL_CCTA and CACS-CL_CCTA using 5% and 15% cut-offs.

Results

The training cohort (n = 38 269) was used for model development and cut-off derivation; the validation cohort (n = 28 340) for model and cut-off validation. Baseline characteristic and diagnostic test results for the training and validation cohorts are shown in Table 1.

Model development

In the training cohort, observed obstructive CAD at CCTA was present in 8068 (21.1%) patients. In the training cohort, the observed prevalence of observed obstructive CAD at CCTA was higher compared with the predicted mean probability by the Basic PTP (13.7%, P < 0.001). Male sex and age, number of risk factors, and CACS increased with observed obstructive CAD prevalence at CCTA (see Supplementary data online, Table S1 and Figure S1).

The RF-CL_CCTA model was developed based on logistic regression analyses including risk factor categories, sex, age, and symptom typicality, with the CACS-CL_CCTA additionally including CACS to the clinical likelihood estimation (Figure 1). For both the Basic PTP, the RF-CL_CCTA, and CACS-CL_CCTA models, the prevalence of observed obstructive CAD at CCTA increased with increasing clinical likelihood.

In the training cohort, the RF-CL_CCTA and CACS-CL_CCTA models showed superior calibration and discrimination against observed obstructive CAD at CCTA compared with the Basic PTP predictive model, with the CACS-CL_CCTA model yielding overall highest discrimination (see Supplementary data online, Figure S2).

In the training cohort, diagnosed obstructive CAD at ICA was present in 3057 (8.0%) patients. An ultra-low clinical likelihood group was identified as patients with <5% RF-CL_CCTA and CACS-CL_CCTA in whom the prevalence of diagnosed obstructive CAD at ICA was <1% (Figure 2, Supplementary data online, Table S3). Prevalences of diagnosed obstructive CAD at ICA below 5%, 15%, and 50% were observed for RF-CL_CCTA and CACS-CL_CCTA cut-points of <17%, <35%, and <82%, respectively.

Model validation

In the pooled validation cohort, observed obstructive CAD at CCTA was present in 6443 (22.7%) patients. For both the Basic PTP, the RF-CL_CCTA, and CACS-CL_CCTA models, the prevalence of observed obstructive CAD at CCTA increased with increasing clinical likelihood (see Supplementary data online, Table S2 and Figure S3).

In the pooled validation cohort, the Basic PTP underestimated the prevalence of observed obstructive CAD at CCTA, with the RF-CL_CCTA and CACS-CL_CCTA models showing good and superior calibration. Against observed obstructive CAD at CCTA, the RF-CL_CCTA and CACS-CL_CCTA models had superior discrimination, with the CACS-CL_CCTA model yielding overall highest discrimination. Similar results were present within the respective cohorts founding the pooled validation cohort, with minor variation depending on the prevalence of observed obstructive CAD at CCTA (Figure 3 and Supplementary data online, Figure S4).

At a 5% cut-off for prediction of observed obstructive CAD at CCTA, the Basic PTP, RF-CL_CCTA, and CACS-CL_CCTA models yielded high sensitivities (96–99%) at the cost of low specificities (6–24%) (Table 2). At a 15% cut-off, the RF-CL_CCTA and CACS-CL_CCTA models maintained high sensitivities (89–93%) while sensitivity was markedly impaired for the Basic PTP model (68%). Compared with the 5% cut-off, specificity increased at the 15% cut-off for all models. NRIs at 5% and 15% cut-offs for prediction of observed obstructive CAD at CCTA are shown in Supplementary data online, Table S4.

In the temporal WDHR validation cohort (n = 8844), diagnosed obstructive CAD at ICA was present in 776 (8.8%) patients. The prevalence of diagnosed obstructive CAD at ICA was <1% for patients with <5% RF-CL_CCTA and CACS-CL_CCTA (see Supplementary data online, Table S3), classifying 595 (6.7%) patients at ultra-low risk. Prevalences of observed obstructive CAD at ICA < 5%, <15%, and 50% were observed for RF-CL_CCTA and CACS-CL_CCTA estimates < 20%, <33%, and 82%, respectively.

Discussion

Recommendations for the evaluation of stable patients with new-onset chest pain include the use of current PTP algorithms which are calibrated against mixed ICA/CCTA reference standards, although non-invasive testing and especially CCTA is the recommended index test in such patients. In this study, we propose the use of newly-derived CCTA-only calibrated clinical likelihood models as more appropriate PTP tools in the large group of low risk patients, and refine the CCTA-derived PTP using risk factor- and coronary artery calcium score-weighted clinical likelihood models. In a large-scale global external validation cohort, the novel RF-CL_CCTA and CACS-CL_CCTA models show superior calibration and discrimination of obstructive CAD at CCTA compared with a Basic PTP model. Additionally, we identified an ultra-low clinical likelihood cohort with a prevalence of diagnosed obstructive CAD at ICA < 1% and propose CCTA-calibrated clinical likelihood cut-offs to guide patient management in patients likely to have <1% obstructive CAD on ICA findings.

Clinical likelihood estimation in chronic coronary syndrome

Recognized as a gap in evidence by the 2019 ESC guidelines on chronic coronary syndrome,⁶ the original RF-CL and CACS-CL models were developed as tabulated, simple, and clinically useful tools for improved prediction of obstructive CAD in de novo chest pain patients.⁷ Other advanced models have been proposed^18,19 but their clinical utility is limited by the need for online calculation.

Clinical likelihood and reference standard of obstructive CAD

Originally, the RF-CL and CACS-CL models were calibrated against 50% diameter stenosis at ICA as reference for obstructive CAD.⁷ In general, a reference standard of diagnosed obstructive CAD at ICA has been preferred for pre-test likelihood models as this reference is related to treatment decisions and ultimately revascularization. CCTA overestimates invasively assessed lesion severity, and the positive predictive value of abnormal CCTA is only 40–50% when compared with ICA stenosis. Hence, ICA-calibrated clinical likelihood models are expected to underestimate the prevalence of observed obstructive CAD at CCTA,^7,9 leaving clinicians uncertain as to how to interpret CCTA findings.

The applied Basic PTP model, which was previously endorsed by the 2019 ESC guideline, is a pooled analysis of three studies: the CONFIRM registry,²⁰ the PROMISE trial,²¹ and a study by Reeh et al.²² For model derivation, the reference standard constituted observed obstructive CAD at CCTA by either site- or core lab-reading in the CONFIRM registry and PROMISE trial, respectively, whereas the study by Reeh et al. defined obstructive CAD based on ICA findings. Compared with the novel RF-CL_CCTA and CACS-CL_CCTA models, the Basic PTP model expectedly underestimated the prevalence of observed obstructive CAD at CCTA, with overall impaired calibration (Graphical Abstract) and only yielding similar calibration in the PROMISE trial, which used site-read data,²¹ and the Brazil cohort with an observed low prevalence obstructive CAD at CCTA (14.8%) (Figure 3 and Supplementary data online, Figure S4). Importantly, in PROMISE, core lab-based CCTA reports yielded lower disease prevalence.²³ By the novel RF-CL_CCTA and CACS-CL_CCTA models, we provide an extensively validated tool that, similarly to the ICA-calibrated clinical likelihood models that improve prediction of diagnosed obstructive CAD at ICA,⁷ improve calibration and discrimination of observed obstructive CAD on a clinically relevant CCTA read (Graphical Abstract).

Clinical implications

Decisions on revascularization in patients with chronic coronary syndrome rely on ICA findings often accompanied with invasive or non-invasive functional information, highlighting the importance of calibrating clinical likelihood models against a reference of diagnosed obstructive CAD at ICA. However, given recent trial data,^24,25 the use of CCTA reduces unnecessary invasive catheterizations and catheterization-associated complications. Further, as most de novo chest pain patients are in a lower clinical likelihood subgroup with recommendations for primary testing using CCTA,² clinical likelihood models calibrated against a reference of observed obstructive CAD at CCTA are relevant. Importantly, we present clinical likelihood cut-offs for categories that correlates prevalences of observed obstructive CAD at CCTA and diagnosed obstructive CAD at ICA, identifying an ultra-low clinical likelihood subset of patients with ≤1% probability of invasively assessed stenosis (Figure 2 and Supplementary data online, Table S2). As traditional clinical likelihood cut-offs are based on the observed prevalence of diagnosed obstructive CAD at ICA, these new CCTA-calibrated thresholds can now guide patient management in a guideline-recommended diagnostic workup algorithm utilizing index CCTA.^1,2 Future validation of the ESC guideline-recommended RF-CL and CACS-CL models should utilize the CL model calibrated to the endpoint corresponding to the study-defined reference of obstructive CAD. This is important because the vast majority of validation studies of the CL model (calibrated against invasive angiography) use CCTA-based stenosis as the endpoint,^7,26,27 where the present paper recalibrated these likelihood estimates.

Using the established clinical likelihood cut-off of 15% as predicted by the novel CCTA-calibrated clinical likelihood models (Figure 2), two in five patients with clinical indication for CCTA could be deferred from testing with a predicted prevalence of diagnosed obstructive CAD at ICA ≤ 5% (Figure 2 and Supplementary data online, Table S3). Similarly, using a 30% CCTA-calibrated cut-off, the prevalence of diagnosed obstructive CAD at ICA is ≤15%, and the proportion of patients in whom downstream testing could be deferred is substantial. By a 5% cut-off, specifically specificities are poor using the CCTA-calibrated models (Table 2), which increases to ranges similar to the ICA-calibrated using a 5% likelihood cut-off,⁷ which underscores the interchangeability of the CCTA- and ICA-calibrated models by different model cut-offs based on disease prevalences (Figure 2). Overall, as prognosis in lower likelihood patients is good with reported annualized event rates < 1%,⁸ and contemporary evidence highlights only symptomatic benefit of revascularization compared with guideline-directed medical therapy in most patients,^28,29 the novel RF-CL_CCTA and CACS-CL_CCTA models including their clinically relevant cut-offs seemingly provide safe and tangible evidence to guide referral for CCTA and everyday clinical decision making.

Future direction of clinical likelihood models

Studies suggest that identification of non-obstructive CAD by coronary CTA and initiation of preventive medical therapy reduce event rates in stable chest pain patients.^16,30 However, whether being calibrated against obstructive CAD at ICA or obstructive CAD at CCTA, clinical likelihood models without CACS utilization (i.e. the Basic PTP and RF-CL_CCTA models, alternatively the exercise ECG-weighted clinical likelihood model³¹) can be used to potentially defer testing in patients with non-obstructive coronary lesions. An alternative is the PROMISE Minimal Risk Score which improves discrimination of non-atherosclerotic coronary arteries and has been shown to safely guide testing deferral including data from a randomized trial.^19,25,32 Whether a strategy to identify those patients estimated to be at minimal risk is preferable compared with a risk factor- and CACS-based clinical likelihood estimation is unknown. Moreover, as chest pain complaints presumably are a consequence of myocardial ischaemia, and the presence of ICA stenosis does not necessarily imply impaired coronary flow,^11,13 clinical likelihood models calibrated against impaired myocardial perfusion could be of interest.³³

Limitations

First, all patients were referred for CCTA and selection bias cannot be excluded. Secondly, asymptomatic patients that were not referred for diagnostic testing by their general practitioner or outpatient clinic were not included, and the models potentially over- or underestimate the prevalence of observed obstructive CAD in these patients depending on disease prevalence. Thirdly, patients with severe kidney failure and severe obesity which are not good candidates for CCTA may be underrepresented. Fourthly, the RF-CL_CCTA and CACS-CL_CCTA models were developed and only validated in patients without previously diagnosed CAD, limiting their utility to de novo chest pain patients. Finally, the RF-CL_CCTA and CACS-CL_CCTA models were developed in patients having available information on both CACS and CCTA, and patients who did not undergo CCTA because of an initial CACS examination yielding high values were excluded. However, when including patients with only CACS and no CCTA in the training cohort (n = 1819) and using a combined endpoint of CACS > 400 and the downstream ICA result to surrogate the presence of CCTA-observed stenosis, the prevalence of observed obstructive CAD on CCTA would increase from 21% to 22%, and the effect on model calibration would be negligible.

Conclusions

Novel CCTA-calibrated risk factor- and coronary artery calcium score-weighted clinical likelihood models improve calibration and discrimination of observed obstructive CAD at CCTA.

Acknowledgements

L.D.R. acknowledges support from Danish Cardiovascular Academy (grant number PD5Y-2023001-DCA) which is funded by the Novo Nordisk Foundation (grant number NNF20SA0067242) and The Danish Heart Foundation. S.W. acknowledges support from the Novo Nordisk Foundation Clinical Emerging Investigator grant (NNF21OC0066981). L.B. acknowledges support from the National Medical Research Council (NMRC) of Singapore Centre (grant number CG21APR1006), and the NMRC of Singapore Transitional Award (grant number: TA21nov-0001). All other authors have nothing to declare.

Supplementary data

Supplementary data are available at European Heart Journal - Cardiovascular Imaging online.

Funding

The study was supported by The Danish Heart Foundation (grant no. 15-R99-A5837-22920), the Health Research Fund of Central Denmark Region, and Aarhus University Research foundation. The PROMISE trial was supported by grants from the National Heart, Lung, and Blood Institute (R01HL098237, R01HL098236, R01HL098305, and R01HL098235).

Data availability

No new data were generated or analysed in support of this research.

References

Author notes