Cardiovascular disease risk assessment using a deep-learning-based retinal biomarker: a comparison with existing risk scores

Abstract

Aims

This study aims to evaluate the ability of a deep-learning-based cardiovascular disease (CVD) retinal biomarker, Reti-CVD, to identify individuals with intermediate- and high-risk for CVD.

Methods and results

We defined the intermediate- and high-risk groups according to Pooled Cohort Equation (PCE), QRISK3, and modified Framingham Risk Score (FRS). Reti-CVD’s prediction was compared to the number of individuals identified as intermediate- and high-risk according to standard CVD risk assessment tools, and sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated to assess the results. In the UK Biobank, among 48 260 participants, 20 643 (42.8%) and 7192 (14.9%) were classified into the intermediate- and high-risk groups according to PCE, and QRISK3, respectively. In the Singapore Epidemiology of Eye Diseases study, among 6810 participants, 3799 (55.8%) were classified as intermediate- and high-risk group according to modified FRS. Reti-CVD identified PCE-based intermediate- and high-risk groups with a sensitivity, specificity, PPV, and NPV of 82.7%, 87.6%, 86.5%, and 84.0%, respectively. Reti-CVD identified QRISK3-based intermediate- and high-risk groups with a sensitivity, specificity, PPV, and NPV of 82.6%, 85.5%, 49.9%, and 96.6%, respectively. Reti-CVD identified intermediate- and high-risk groups according to the modified FRS with a sensitivity, specificity, PPV, and NPV of 82.1%, 80.6%, 76.4%, and 85.5%, respectively.

Conclusion

The retinal photograph biomarker (Reti-CVD) was able to identify individuals with intermediate and high-risk for CVD, in accordance with existing risk assessment tools.

Graphical Abstract

The Pooled Cohort Equation (PCE), QRISK3, and modified Framingham Risk Score (FRS) are formal risk assessment tools to guide primary prevention of cardiovascular disease in the USA, UK, and Singapore, respectively. However, these risk assessment tools require a lipid profiling via blood tests. Currently, a non-invasive cardiovascular disease (CVD) risk assessment tool is not available, particularly in stratifying those at intermediate- and high-risk. Reti-CVD is a retinal imaging biomarker developed from a deep-learning algorithm that can predict future CVD risk. In this study, we aim to evaluate the performance of Reti-CVD as a non-invasive triage tool to identify individuals with intermediate- and high-risk for CVD who can potentially benefit from early intervention. In this cross-sectional validation study, we confirmed Reti-CVD’s ability to identify individuals with intermediate- and high-risk for CVD based on their 10-year CVD risk using three different standard risk assessment tools (PCE, QRISK3, and modified FRS) with more than 80% sensitivity and specificity. Our findings demonstrate Reti-CVD’s ability to function as a non-invasive screening tool that can help triage patients according to CVD risk.

Open in new tabDownload slide

Introduction

In the USA, the population of older Americans is not only increasing, but also nearly half of US adults are expected to have cardiovascular disease (CVD) by 2035.¹ An individual’s 10-year CVD risk is estimated using the Pooled Cohort Equation (PCE), which separates people into four risk categories: low (5%), borderline (5–7.4%), intermediate (7.5–19.9%), or high (≥20%).² For individuals with intermediate-risk for CVD, it is important to consider risk-enhancing factors that may increase their actual risk for CVD.³ Such individuals often benefit the most from statin therapy.

In the UK, around one in nine people live with CVD.⁴ A formal risk assessment tool, QRISK3, was developed using data from a cohort of 1.28 million individuals, and it is currently recommended by the National Institute for Health and Care Excellence for CVD risk assessment.⁵ The current recommendation for initiating statins for primary CVD prevention and antihypertensives drugs for stage 1 hypertension is a calculated QRISK3 score of 10%.^6,7 Therefore, the benchmark of 10% for 10-year CVD risk is crucial in determining clinical intervention.

The modified Framingham risk score (FRS) is a formal risk assessment tool utilized in Singapore where individuals are classified as low (<10%), intermediate (10% to 20%), or high-risk (≥20%) for CVD.⁸ These risk categories inform various treatment guidelines in Singapore, especially with regards to recommended LDL cholesterol levels.

These standard CVD risk assessment tools have provided physicians with a way to treat patients pre-emptively. Though each CVD risk assessment tool has its benefits and drawbacks based on the sample size, follow-up period, and number of risk factors utilized, the most concerning issue of these three CVD risk assessment tools is the fact that they all utilized a cohort of patients of Western descent. With new knowledge of the different risk factor profiles of different ethnicities, these risk assessment tools are not the most accurate for all populations.⁹

Deep learning comprised neural networks, which are adept at computerized visual perception and image recognition and has thrived in image-centric medical field.¹⁰ Previous studies have shown the potential of artificial intelligence in personalized CVD risk stratification. A previous study showed how machine-learning outperformed CVD risk calculator in a 13-year follow-up dataset from the Multi-Ethnic Study of Atherosclerosis.¹¹ Other studies utilized various variables to predict and stratify CVD risk using deep-learning algorithms and showed deep-learning method was superior than conventional CVD risk calculators or machine-learning method in prediction of CVD risk, indicating that deep learning has potential to assess personalized risk more accurately.¹²

Many of these studies predict and stratify CVD risk using algorithms that utilize known risk factors for CVD such as age, gender, hypertension, and carotid ultrasound images. However, the retina can manifest early stages of CVD through microvascular abnormalities. Retinal photography is a simple, effective, and non-invasive imaging tool that provides fast and accurate information on the human vasculature that may not be clearly visible to the human eye especially in recent years with the advent of deep learning.^13–15 In our previous study, we developed a deep-learning-based CVD risk stratification system using Reti-CVD, which allowed us to stratify future CVD risk in the UK Biobank and a diversified Asian population.¹⁶ Although Reti-CVD showed great promise in predicting future CVD risk in various ethnic groups including Koreans, Malays, Indians, Chinese, and the British,¹⁶ Reti-CVD has not been validated in identifying at-risk groups according to existing CVD risk stratification clinical guidelines.

We hypothesize that Reti-CVD can identify people with intermediate- and high-risk for CVD in accordance with three existing standard CVD risk assessment tools: PCE, QRISK3, and modified FRS. We aim to validate this hypothesis using two population-based studies from the UK and Singapore and validate Reti-CVD’s potential to be used as a non-invasive triage tool to screen individuals who can benefit from earlier preventative CVD intervention.

Methods

Ethics statement

This cross-sectional study was deemed exempt from institutional review board review by the SingHealth Centralised Institutional Review Board. This study adhered to the tenets of the Declaration of Helsinki. Written informed consent was obtained from the participants of the original studies.^17,18

Study population

We used two large studies: the UK Biobank, a prospective community-based cohort in the UK,¹⁹ and Singapore Epidemiology of Eye Diseases (SEED) study, a population-based study in Singapore.²⁰

In the UK Biobank, we excluded participants as follows: (i) with duplicated retinal photographs (n = 18 423); (ii) who had type 1 diabetes (n = 288); (iii) with pre-existing CVD at baseline (n = 7624); (iv) with poor-quality photographs (n = 11 115); and (v) who were <40 years old (n = 1) (Figure 1A). Pre-existing CVD was defined as previous history of coronary heart disease, other heart diseases, stroke, transient ischaemic attack, peripheral arterial disease, or cardiovascular surgery, and patients who have undergone cardiovascular procedures based on the Classification of Interventions and Procedures version 4.¹⁶ A total of 48 260 participants, representing the general population without a history of CVD, were included for analysis. Retinal photographs included in the study were taken using the Topcon 3D OCT-1000 Mark II (Topcon Corporation) between 2009 and 2010.

In the SEED study, we excluded participants as follows: (i) with missing variables at baseline (n = 596) and (ii) with pre-existing CVD at baseline (n = 667) (Figure 1B). Pre-existing CVD was defined as previous history of cardiovascular disease. A total of 6810 participants (2548 Chinese, 1976 Indian, and 2286 Malay), representing the general population without a history of CVD, were included for analysis. Retinal photographs included in the study were taken using the Canon CR-1 Mark-II Nonmydriatic Digital Retinal Camera (Canon Corporation) between 2004 and 2011.

Definition of intermediate- and high-risk

We defined ‘intermediate- and high-risk’ as those patients whom CVD risk management may be a priority based on PCE, QRISK3, and modified FRS. For each individual in the UK Biobank, we estimated CVD risk according to the PCE, in accordance with the American College of Cardiology/American Heart Association guidelines,² and QRISK3 score using R package.²¹ For each individual in SEED, the modified FRS was calculated.⁸ We then compared the following groups: (i) PCE-based 10-year CVD risk ≥7.5% vs. < 7.5% (i.e. intermediate- and high-risk vs. low- and borderline-risk group) and (ii) QRISK3/modified FRS-based 10-year CVD risk ≥10% vs. < 10% (i.e. intermediate- and high-risk vs. low-risk group).

New retinal-based cardiovascular disease risk stratification system

Our previous studies detail the development and validation of Reti-CVD¹⁶ Briefly, the Reti-CVD score was developed based on data from retinal photographs and CT-measured coronary artery calcium (CAC) via a deep-learning algorithm. The probability scores ranged from zero to one, with a high value indicating a higher probability of CAC presence. To generate the saliency maps, we used guided backpropagation, and aggregated analysis was done. Detailed methods and updates of Reti-CVD score were described in Supplementary material online, eDocument S1.

In the present study, we proposed binary CVD risk groups based on preliminary analysis to maximize performance in detecting the intermediate- and high-risk CVD groups in each study using the receiver operating characteristic (ROC) curve and the Youden index. We divided participants into low-risk and high-risk according to Reti-CVD scores. We used this proposed binary risk group based on Reti-CVD scores to further validate Reti-CVD’s ability to detect intermediate- and high-risk groups in each study. Detailed ROC analysis and methods were provided in Supplementary material online, eDocument S2. In addition, we used covariate-adjusted ROC analysis²² to adjust for potential covariates affecting the CVD risk in ROC analysis (see Supplementary material online, eDocument S3).

At-risk group and cardiovascular risk factors

The other variables used in this study were defined as follows. In the UK Biobank, type 2 diabetes was defined based on (i) medical history; and/or (ii) glucose ≥5.5 mmol/L. In the SEED, diabetes was defined based on (i) a random blood glucose ≥200 mg/dL (11.1 mmol/L); (ii) self-reported physician-diagnosed diabetes; (iii) glycated hemoglobin A1c (HbA1c) ≥ 6.5%; and/or (iv) use of glucose-lowering medication in the SEED.²³ In both studies, individuals with high cholesterol, diabetes, and hypertension were identified through self-reporting via a baseline assessment questionnaire. Similarly, smoking status was self-reported by individuals and categorized into either ‘lifetime smoker’ or ‘never.’ Overweight was defined as body mass index of ≥30 kg/m² in the UK Biobank and ≥25 kg/m² in the SEED.

Statistical analysis

Descriptive statistics were provided for the study population in the UK Biobank and the SEED. Binary 2 × 2 Confusion matrix was provided in each study for the study population along with a breakdown by gender and specific at-risk groups including lifetime smokers and overweight individuals. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were determined to evaluate Reti-CVD’s ability to detect intermediate- and high-risk groups in all participants and specific at-risk groups. In addition, ethnic-specific metrics were calculated in the SEED study. Analyses were done using P < 0.05 as the significance level, Stata/MP version 14.0. For sensitivity analysis, we repeat the analysis using SEED study on PCE-based risk groups.

Results

Table 1 details the clinical characteristics of the participants in the UK Biobank and the SEED categorized by their CVD risk according to PCE (low and borderline-risk vs. intermediate- and high-risk) QRISK3 (low vs. intermediate- and high-risk) in the UK Biobank and modified FRS in the SEED.

Among 48 260 included participants from the UK Biobank, 20 643 had ≥7.5% 10-year CVD risk according to PCE, and 7192 had ≥10% 10-year CVD risk according to QRISK3. Among those classified as intermediate- and high-risk, 82.7% (17 069/20 643) and 82.6% (5939/7192) belong to the high-risk group based on the Reti-CVD binary risk stratification system. Among 8770 included participants from the SEED study, 3799 had ≥10% 10-year CVD risk according to modified FRS and were classified as intermediate- and high-risk. Among these 3799 people classified as intermediate- and high-risk, 82.1% (3119/3799) belong to the high-risk group based on the Reti-CVD binary risk stratification system. In both studies, CVD risk factors such as old age, gender, hypertension, diabetes, lifetime smoking status, and being overweight were more prevalent in intermediate- and high-risk groups than in low-risk groups.

Aggregated saliency maps indicated that the highlighted area along the arcade vessels was more prominent in images of Reti-CVD high risk group than in those of Reti-CVD low risk group (see Supplementary material online, eFigure S1).

Table 2 details the number of participants classified by Reti-CVD’s binary risk groups (low vs. high) and by different standard CVD risk assessment tool’s two risk groups (low/borderline vs. intermediate and high) in all participants with a breakdown by gender. Repeated analysis according to specific at-risk factors (smoking status and overweight status) are provided in Supplementary material online, eTable S1.

Table 3 summarizes Reti-CVD’s performance in identifying intermediate- and high-risk groups (defined by the different formal CVD risk assessment tools). According to PCE and QRISK3, the overall proportion of individuals in the UK Biobank categorized as intermediate- and high-risk was 49.0% and 15.0%, respectively. Reti-CVD identified PCE-based intermediate- and high-risk groups with a sensitivity, specificity, PPV, and NPV of 82.7%, 87.6%, 86.5%, and 84.0%, respectively. Reti-CVD identified QRISK3-based intermediate- and high-risk groups with a sensitivity, specificity, PPV, and NPV of 82.6%, 85.5%, 49.9%, and 96.6%, respectively. In the SEED study, the overall prevalence of people classified as intermediate- and high-risk was 43.0%. Reti-CVD identified intermediate- and high-risk groups according to the modified FRS with a sensitivity, specificity, PPV, and NPV of 82.1%, 80.6%, 76.4%, and 85.5%, respectively. Repeated analysis according to specific at-risk groups (lifetime smoking status and overweight status) and age are provided in Supplementary material online, eTables S2 and S3.

Table 4 summarizes Reti-CVD’s performance in identifying intermediate- and high-risk groups (defined by the different formal CVD risk assessment tools) in each ethnic group from the SEED study. The prevalence of intermediate- and high-risk groups was 26.0% in Singaporean Chinese, 56.0% in Singaporean Indian, and 48.0% in Singaporean Malay. Sensitivity of Reti-CVD in identifying intermediate- and high-risk groups was 79.7% in Singaporean Chinese, 83.2% in Singaporean Indian, and 82.1% in Singaporean Malay. The NPV of Reti-CVD in identifying intermediate- and high-risk groups was especially high with 99.3% in Singaporean Chinese females. Repeated analysis according to specific at-risk groups (lifetime smoking status and overweight status) is provided in Supplementary material online, eTable S4.

Sensitivity analyses that applied PCE-risk group on SEED study also showed results similar to those of the main analysis (see Supplementary material online, eTables S5 and S6).

Discussion

In this study, we used Reti-CVD to identify individuals at intermediate- and high-risk for CVD according to PCE (10-year CVD risk ≥7.5%) and QRISK3/modified FRS (10-year CVD risk ≥10%) in the UK and Singapore. The retinal imaging CVD biomarker, Reti-CVD, adequately identified individuals of PCE-based intermediate- and high-risk with a sensitivity of 82.7% in the UK Biobank, individuals of QRISK3-based intermediate- and high-risk with a sensitivity of 82.6% in the UK Biobank, and individuals of modified FRS-based intermediate- and high-risk with a sensitivity of 79.7% in Chinese, 83.2% in Indian, and 82.1% in Malay in Singapore.

First, Reti-CVD can increase CVD risk screenings and make the process more efficient. By identifying high-risk patients earlier and encouraging patients to undergo a full health check-up for further evaluation, Reti-CVD can support national health screening programs such as the ‘UK NHS Health Check’ and ‘Singapore Screen for Life.’ In the UK, this could potentially boost the compliance rate for NHS health screening, which has been estimated to be less than 50%.²⁴ Moreover, the non-invasive nature of Reti-CVD can further promote public screening compliance as it avoids the need for patients to undergo the use of needles for fasting glucose and lipid levels for routine labs. This would encourage previously reluctant patients who have trypanophobia to seek health screening examinations. Given the large delays in phlebotomy due to resource limitations from lack of trained personnel to insufficient equipment, Reti-CVD provides another accurate way to assess CVD risk in an expedited fashion. Furthermore, the non-invasive nature of Reti-CVD would be more cost-effective for patients as it simply requires an image of the retina that requires minimal personnel. Of note, these retinal cameras already exist in many optometric offices that can be incorporated into pre-existing healthcare system to further offset the high initial investment.

Second, as Reti-CVD is an imaging biomarker that can be obtained on routine eye examinations, the opportunistic nature of this CVD risk screening may help include ‘hard to reach’ populations in CVD risk assessment. For example, the convenience of accessing Reti-CVD in neighbourhood optical shops could increase participation rates and complement the sub-optimal NHS Health Check participation rate.²⁴ This makes Reti-CVD compatible for use in many parts of the UK that are currently providing various ophthalmic services (e.g. NHS Sight Test providers, acute eye clinics, diabetic retinopathy screening programs, etc.) In fact, vision loss is highly correlated with social determinants of health.²⁵ In vulnerable populations, these ‘hard to reach’ patients can be accessed more routinely by optometrist as these patients are more likely to seek ophthalmic care for glasses and vision concerns. Furthermore, third, since Reti-CVD simply requires an image of the retina, this opportunistic CVD screening can be utilized to retrospectively screen the general public using stored retinal photographs. Moreover, in an era where telemedicine is becoming the standard of care, especially with the COVID-19 pandemic, this retrospective CVD screening in conjunction with telemedicine could improve patient care. For example, patients could transmit their latest Reti-CVD results during televisits to physicians who can use this information to guide patients on whether or not to seek a more detailed CVD risk assessment. With such easily accessible screening opportunities, healthcare professionals can promote a paradigm shift in healthcare from one that focuses on disease treatment to one that focuses on preventative measures such as Life’s Essential 8 and American Heart Association’s recent guidelines.²⁶ By identifying patients with CVD risk through the retina, patients can be treated pre-emptively for CVD with statins. Studies have demonstrated how statin use reduces the risk of CVD events when patients have an elevated 10-year CVD risk.²⁷ Moreover, depending on patient-specific risk factors, physicians can address these risk factors through lifestyle modifications and medications for hypertension and diabetes. Additional research on trends by comparing older and more recent retinal scans will add another layer of predictive power to our AI algorithm, which would enhance its ability to evaluate CVD risk.

Fourth, given Reti-CVD’s ability to predict future CVD risk in the UK Biobank and diverse Asian populations, adults who are high-risk according to Reti-CVD but have <10% CVD risk according to formal risk assessment tools (i.e. false positive) could obtain life-saving early interventions, which may be extremely cost-effective for providers and patients since tertiary prevention of CVD is significantly more expensive than primary and secondary prevention of CVD.¹⁶ In our explorative analysis using QRISK3, the CVD incidence was 8.6 per 1000 person-years in this false positive group, which is more than double compared to adults who are low-risk according to Reti-CVD and have <10% CVD risk according to formal risk assessment tools (3.3 per 1000 person-years) in the UK Biobank. Reti-CVD has the potential to serve as a highly desirable risk detector that is non-invasive, scalable, and specific.

The strengths of this study demonstrate the efficacy of deep-learning-based imaging biomarkers in accordance with current clinical guidelines of standard CVD risk assessment tools by using large-scale population-based studies in two different countries. Moreover, this study also demonstrates the potential of using the retina to assess CVD risk thereby opening many possibilities in addressing CVD preventatively by creating another potential cost-effective office-based test for patients as noted above. However, since Reti-CVD is based on a deep-learning algorithm, there are inherent limitations to be aware of. Due to the nature of testing and training deep-learning algorithms through datasets, the accuracy of Reti-CVD scores is dependent upon the dataset that was used to train the algorithm and open to biases that occur while compiling the dataset. As we excluded those who had missing clinical variables, if the 10-year CVD risk of excluded individuals differs significantly form those included, it may affect the actual performance of Reti-CVD score. Moreover, many datasets stem from voluntary studies that often attract an accessible population that is not always generalizable. Therefore, Reti-CVD must be used with caution when applying it to new sub-populations or across different ethnicities. Thus, Reti-CVD scores used in this study may not be applicable globally. Moving forward, it is always possible to find more optimal Reti-CVD cut-off values for different populations, especially if there exist population-based eye studies, including retinal photographs and blood tests. To further validate Reti-CVD and decrease potential bias, we hope to evaluate Reti-CVD using cohort studies in other countries such as the USA in the future. The effectiveness of Reti-CVD among those with eye disease was not extensively evaluated. In our explorative analysis, we confirmed there were no differences in area under ROC curve (AUCs) between those without cataract and with cataract in SEED study (AUCs, 0.824 and 0.817, respectively, P = 0.568). In addition, it has been acknowledged that other risk factors such as age and/or gender can affect test outcomes in AUCs. Thus, we provided covariate-adjusted AUCs in the supplementary materials online, eDocument S3. Moreover, we excluded poor quality images during the development and validation of Reti-CVD. The quality of retinal photographs in clinical settings may vary and may pose challenges for real-world implementation. Though completely replacing traditional means of assessing CVD risk through current standard risk assessment models with Reti-CVD may be premature, this study demonstrates the potential of deep-learning in assessing CVD risk.

In conclusion, a new deep-learning and retinal photograph-derived CVD biomarker, Reti-CVD, could effectively identify individuals with intermediate- and high-risk for CVD in accordance with different national guidelines. As a CVD risk triage tool, Reti-CVD has the potential to not only make CVD risk assessment more affordable and accessible, but also benefit patients through early interventions and awareness.

Supplementary material

Supplementary material is available at European Heart Journal—Digital Health online.

Author contributions

T.H.R. conceptualized the study. H.K. and G.L. developed the algorithm. J.K.Y. and T.H.R. wrote the manuscript and researched its contents. All authors provided comments to improve the manuscript. A.B. verified the dataset.

Funding

This project is supported by the Agency for Science, Technology and Research (A*STAR) under its RIE2020 Health and Biomedical Sciences (HBMS) Industry Alignment Fund Pre-Positioning (IAF-PP) Grant No. H20c6a0031. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the A*STAR.

Data availability

The UK Biobank test dataset was obtained from UK Biobank (application number 68 428). Data cannot be shared publicly due to the violation of patient privacy and the absence of informed consent for data sharing. Data from Singapore are available to researchers who meet the criteria for access to confidential data; requests should be made to Prof. Ching-yu Cheng ([email protected]).

References

Author notes