A framework of deep learning networks provides expert-level accuracy for the detection and prognostication of pulmonary arterial hypertension 

Abstract

Aims

To test the hypothesis that deep learning (DL) networks reliably detect pulmonary arterial hypertension (PAH) and provide prognostic information.

Methods and results

Consecutive patients with PAH, right ventricular (RV) dilation (without PAH), and normal controls were included. An ensemble of deep convolutional networks incorporating echocardiographic views and estimated RV systolic pressure (RVSP) was trained to detect (invasively confirmed) PAH. In addition, DL-networks were trained to segment cardiac chambers and extracted geometric information throughout the cardiac cycle. The ability of DL parameters to predict all-cause mortality was assessed using Cox-proportional hazard analyses. Overall, 450 PAH patients, 308 patients with RV dilatation (201 with tetralogy of Fallot and 107 with atrial septal defects) and 67 normal controls were included. The DL algorithm achieved an accuracy and sensitivity of detecting PAH on a per patient basis of 97.6 and 100%, respectively. On univariable analysis, automatically determined right atrial area, RV area, RV fractional area change, RV inflow diameter and left ventricular eccentricity index (P < 0.001 for all) were significantly related to mortality. On multivariable analysis DL-based RV fractional area change (P < 0.001) and right atrial area (P = 0.003) emerged as independent predictors of outcome. Statistically, DL parameters were non-inferior to measures obtained manually by expert echocardiographers in predicting prognosis.

Conclusion

The study highlights the utility of DL algorithms in detecting PAH on routine echocardiograms irrespective of RV dilatation. The algorithms outperform conventional echocardiographic evaluation and provide prognostic information at expert-level. Therefore, DL methods may allow for improved screening and optimized management of PAH.

Graphical Abstract

General overview of the two pronged study design aimed at establishing the utility of DL algorithms in detecting pulmonary arterial hypertension and automatically predicting mortality in this setting.

Open in new tabDownload slide

See the editorial comment for this article ‘Using deep learning to diagnose pulmonary hypertension’, by P. van der Bijl and J. J. Bax, https://doi.org/10.1093/ehjci/jeac148.

Introduction

Patients with pulmonary arterial hypertension (PAH) represent a challenging and heterogenous cohort with high morbidity and poor prognosis if left untreated. Definitive diagnosis of the condition requires right heart catheterization to quantify pulmonary vascular resistance and pressures. However, as PAH occurs sporadically in various medical conditions, including connective tissue disease (CTD), human immunodeficiency virus infection, congenital heart disease (CHD), and presenting symptoms are non-specific, the suspicion of PAH must be raised early in the community. Therefore, widely available non-invasive tools for upfront screening would be optimal to enable timely diagnosis of PAH.¹

Transthoracic echocardiography has emerged as the mainstay for screening in this setting and its diagnostic role has previously been highlighted.² Although commonly available, the sensitivity of echocardiography in detecting PAH relies on the presence of sufficient tricuspid regurgitation to estimate right ventricular (RV) systolic pressures (RVSP) using the simplified Bernoulli equation.³ This information is generally supplemented by other indirect signs, e.g. RV dimensions, morphology, and function. The sensitivity and specificity of this approach remains limited, with reported pooled estimates of 83 and 72%, respectively, even in experienced hands.^3,4 Given the non-specific clinical presentation and the challenges of non-invasive diagnosis, it is not surprising that the mean time from symptom onset to definitive diagnosis of PAH has been reported to be 47 months.⁵

Transthoracic echocardiography, beyond being a diagnostic tool for PAH, is also an established source of prognostic markers in this setting.^6,7 It allows for quantification of cardiac chamber dimensions and function, while periodic transthoracic echocardiographic imaging remains the main modality for detecting complications and guiding therapy in this population.⁸

Given the rapid progress in the field of machine and deep learning (DL) neuronal networks,^9–12 we hypothesized that tailored DL algorithms would be able to assist with the diagnosis of PAH and provide prognostic information that compares favourably with traditional measurements obtained by expert echocardiographers.

Methods

All patients underwent a structured echocardiographic investigation between 2005 and 2018. Patients were included from two tertiary centres for CHD and pulmonary hypertension, namely the Royal Brompton Hospital London, UK and the University Hospital Münster, Germany. Therefore, within the CHD cohort all patients without a biventricular circulation, primary valvular disease (other than secondary tricuspid regurgitation) and with associated large shunt lesions (e.g. ventricular septal defects or uncorrected atrioventricular septal defects) were excluded. Patients with PAH were included only if other relevant structural abnormalities were absent. The diagnosis of PAH was made based on invasive catheter assessment according to the relevant guidelines (mean pulmonary arterial pressure >25 mmHg and wedge pressure ≤ 15 mmg) in all patients.¹⁸ In addition, a consecutive series of patients with atrial septal defects (ASD) or repaired tetralogy of Fallot (ToF) and right ventricular dilatation (defined as an RV inflow diameter above 40 mm) was recruited alongside individuals with structurally normal hearts and no evidence of cardiac dysfunction who served as normal controls. All echocardiographic recordings were performed by experienced operators according to internal standards in line with guideline recommendations^13,14 and measurements were collected retrospectively from echocardiographic reports. Recordings of the apical 4-chamber and parasternal short axis 2-chamber view were retrieved from the electronic hospital systems. The echocardiographic DICOM cine loops were converted to a bitmap format and split into individual frames using a dedicated MatLab (Version R2018a) program. Patient identifying information and metadata were removed before analysis. Retrospective analysis of the de-identified echocardiographic recordings has been approved by the local ethics committee. For survival analyses, data on overall mortality were retrieved from the Office for National Statistics, which registers all UK deaths. As this represents a retrospective study based on data collected for routine clinical care and administrative purposes (UK National Research Ethics Service guidance), individual informed consent was not required.

The Graphical Abstract shows the overall study design as well as the two aims of the study, detection of PAH, and automatic prognostication in the setting of PAH.

Model architecture and training for diagnosis of pulmonary hypertension

Individual echocardiographic frames were split into a training/validation and a test set, ensuring that all frames from one individual subject were in the same subset with no frames from subjects used for model training included in the subsequent model testing. For all frames, the resolution was reduced to a 164 × 164 pixel greyscale image. Image augmentation with random application of rotations (±10°), width and height shifts (10%), as well as shears and zoom (up to 10 and 5%, respectively) were applied to the frames at run-time during training.

The underlying DL network is illustrated in Figure 1. The network consists of two connected 2D-convolutional subnetworks with a series of convolutions layers (3 × 3 size), max pooling layers (2×) and fully connected layers. In addition, drop-out layers are included to avoid overfitting. Rectified linear units were used as activation functions. One of the subnetworks accepts image information from the apical 4-chamber view, while an image (from the same individual) obtained in a parasternal short axis view is provided as input for the second subnetwork. The output from the two convolutional subnetworks is merged with the echocardiographically estimated RVSP (Bernoulli equation based tricuspid jet velocity derived pressure gradient plus estimated right atrial pressure). The right atrial pressure was estimated based on assessment of the inferior vena cava diameter and collapsibility according to published recommendations.¹⁵ The numeric data for RVSP were rescaled to an interval ranging from 0 to 1 by dividing pressure estimates (in mmHg) by 100 mmHg. Thus, a pressure of 64 mmHg would correspond to a value of 0.64. Pressure values >100 mmHg were set to 1 and patients without RVSP values available were assigned an RVSP value of 0. The amalgamated information was presented to a softmax-classifier layer providing probabilities for the three possible diagnoses. Custom-made generator functions were used to provide input data from the same individual at run time.

Model architecture and training for cardiac chamber segmentation

Cardiac chamber segmentation was performed based on the U-Net architecture implemented in R/Keras as described in detail previously.^16,17 Further details on the network setup as well as the assessment of ventricular geometry in the parasternal short axis view is provided in the Supplementary data online. RV inflow diameter was measured as the distance between the lateral and septal insertion point of the tricuspid valve as illustrated in Figure 3.

Statistical analysis

Descriptive data are shown as median and interquartile range (IQR) corresponding to the 25th and 75th percentile of the data distribution. Categorical variables are given in absolute numbers and percentages. Comparisons between groups were performed by using unpaired t-tests, Mann–Whitney U-tests, and χ² tests depending on data type and the presence of normal distribution, respectively. To assess the association between conventional echocardiographic and DL-based parameters regarding prognosis uni- and multivariable Cox-proportional hazard survival models were fitted after testing the proportional hazards assumption (by assessing the relationship between scaled Schoenfeld residuals and survival time). Multivariable stepwise Cox-proportional models were built including those parameters which were significant on univariable analysis. Redundant or highly correlated parameters from the same cardiac chambers were not included in the model. To illustrate the prognostic value of DL parameters, we constructed a model utilizing the significant DL-based echocardiographic parameters and contour plots of the estimated 2-year survival for various combinations are shown. Comparisons between the discriminatory quality of different parameters were made with the use of the CompareC R-package. For all analyses R version 3.5.1 was used. For all analyses a two-sided P-value of <0.05 was considered statistically significant.

Results

Overall, 450 patients with PAH [median age 59 (IQR 46-69) years, 67% female], 308 patients with RV dilatation without PAH (201 repaired TOF and 107 ASD patients), and 67 normal controls were included in the study. Table 1 provides an overview of demographic, clinical, and echocardiographic information of the three groups. In addition, it shows invasive haemodynamic data from the entire PAH cohort. Additional clinical and echocardiographic data stratified by PAH survival status are presented in Table A (Supplementary data online, Onlineappendix). Regarding the clinical classification of PAH,¹⁸ we included 150 patients with idiopathic PAH (Group 1.1.), 4 patients with heritable PAH (Group 1.2.), 178 patients with CTD (Group 1.4.1.), 90 patients with CHD without structural abnormalities (Group 1.4.4), 5 patients with human immunodeficiency virus related PAH (Group 1.4.2.), 5 patients with drugs or toxin induced PAH (Group 1.3.), and 18 patients with portal hypertension related PAH (Group 1.4.3).

In total, 21 417 apical 4-chamber frames from PAH patients, 12 796 frames from non-PAH patients with non-PAH RV dilatation and 11 020 frames from normal controls were available and included in the analyses. In addition, 4917 parasternal short axis view frames from normal controls, 12 524 frames from PAH patients, and 9033 frames from non-PAH RV dilatation patients were available for analysis.

Identification of PAH patients

The ability to diagnose PAH based on tricuspid regurgitation jet velocity estimates of RVSP alone was limited by the number of subjects with available and traceable Doppler spectra. While 84% of PAH patients [n = 380, estimated median RVSP 62 (IQR 46-80) mmHg] had adequate tricuspid regurgitation jet velocity measurements, this was true for 65% of non-PAH related RV dilatation [n = 199, estimated median RVSP 31 (IQR 25-38) mmHg] patients and only 34% of normal controls [n = 23, estimated median RVSP 26 (IQR 21.5-30.75) mmHg]. As a consequence, no definitive assessment of PAH status was possible on tricuspid regurgitation alone in 218 patients (27.0% of study subjects) (Figure 2A) and the overall accuracy, sensitivity, and specificity of this approach was only 58.5, 68.5, and 91.6%, respectively (80.6, 81.6 and 86.0% in patients with available tricuspid regurgitation spectra). In addition, 18.4% of (invasively confirmed) PAH patients with tricuspid regurgitation spectra had RVSP estimates, below 40 mmHg. In contrast, the DL-based approach trained on 53 832 frames and validated on 26 474 frames not used for training, achieved an on-frame accuracy of 95.0% in the test set. On a per patient basis, all normal controls and PAH patients were correctly classified, while two patients with non-PAH RV dilatation, elevated RV pressure, and poor image quality were assigned to the PAH group, yielding an overall accuracy of 97.6% and a sensitivity of detecting PAH of 100% (Figure 2B and C).

To illustrate anatomic structures contributing to the diagnosis of PAH, activation maps of the intermediate network layers were calculated, as previously described (Supplementary data online, Figure A).¹⁹ To this end, the gradients flowing into the final 3 downstream convolutional layers of the classification networks (apical 4-chamber and parasternal 2-chamber subnetwork) were used to produce localization maps, highlighting regions in the input image that are important for predicting the diagnosis. The Figure illustrates that the algorithm activates on relevant myocardial structures, including the RV, the interventricular septum, and right atrial structures.

Cardiac chamber segmentation

The DL models trained for segmentation of the cardiac chambers in the parasternal 2-chamber and apical 4-chamber view achieved a good accuracy of area detection, with IOU scores above 97% for all chambers in the test dataset. The mean area variability comparing ground truth masks with predicted results was below 3% on average for all chambers assessed. Detailed results of the area segmentation results are presented in Supplementary data online, Table B. Based on the results of DL-based segmentation, chamber areas, contour length, and chamber dimensions were calculated frame by frame throughout the cardiac cycle, as illustrated in Figure 3. Supplementary data online, Table A presents the results of the analysis, alongside the results of the calculation of LV and RV eccentricity indices and septal curvature. Per frame analysis time on an Intel i7/Nvidia GeforceGTX1070-equipped commercial laptop was 110 milliseconds on average.

Automatic prognostication in patients with PAH

The prognostic value of automatically derived measures of cardiac chamber dimensions and ventricular function was evaluated and compared with manually obtained echocardiographic measures. By design, the follow-up spanned the time period between last transthoracic echocardiogram and death or end of follow-up. Over a median follow-up of 2.0 years (IQR 0.4–2.7 years), 196 patients died. On univariate Cox-proportional hazard analysis, various DL-derived parameters were found to be significantly related to all-cause mortality. As illustrated in Supplementary data online, Table C, this included right atrial (RA) area, right ventricular (RV) area, left ventricular (LV) area, RV fractional area change, basal (inflow), and LV eccentricity index. The prognostic value of these DL-derived parameters was compared with that of their manually quantified counterpart, where available. On direct c-value analysis, right atrial area, RV inflow diameter, and RV fractional area change as assessed by the DL algorithm were statistically non-inferior to their manually quantified counterparts (P-value between 0.47 and 0.72). A multivariable prognostic model based on significant DL-derived echocardiographic parameters was compared with a model based entirely on manually obtained echocardiographic parameters. On multivariable stepwise Cox-proportional hazards analysis, RA area, and RV fractional area change emerged as significant independent variables, both in the manually measured variable model and the DL-derived model (Table 2). The discriminatory ability of both multivariable models was similar, with c-values of 0.67 and 0.65, respectively. Figure 4 is based on the results of the DL-based multivariable model and illustrates that a combination of right atrial area and automatically measured RV fractional area change can stratify patients into high- and low-risk groups.

Discussion

Our study demonstrates, for the first time, the utility of a framework of DL algorithms for automatically detecting PAH based on raw echocardiographic data acquired at an expert centre, incorporating estimated RV systolic pressures when available. The DL algorithm was non-inferior to conventional echocardiographic evaluation in detecting PAH. In addition, we show the ability of DL algorithms to assess prognosis in PAH patients, with a predictive power comparable to models utilizing manually measured echocardiographic features by expert investigators.

Timely diagnosis of PAH remains a major clinical challenge, and has significant implications in terms of patient outcomes, contributing to unnecessary morbidity and potentially mortality.²⁰ Superior outcomes have been described in individuals identified early by an active screening program compared with patients enrolled from routine practice.²¹ This observation is consistent with the results of large multicentre studies, showing the beneficial effect of disease targeting PAH therapy on prognosis and symptoms in PAH patients.^22–24 The reality is, however, that many patients continue to experience delays in diagnosis and, thus, not fully benefit from early PAH therapy. Even in the current era, it has been estimated that the time from first symptom onset to invasive diagnosis and confirmation of PAH may be as long as 47 months.⁵ Transthoracic echocardiography remains the cornerstone of non-invasive ambulatory screening for PAH, and is included in diagnostic algorithms (e.g. the DETECT algorithm for systemic sclerosis).^2,20 Tricuspid regurgitant jet (TR) velocity represents the main criterion for the echocardiographic identification of PAH, but TR is often absent or not traceable and may not accurately reflect invasive pressures.³ The current feasibility study was aimed at developing a DL network that accepts estimated RV systolic pressure values (optionally, when available) and apical 4-chamber and parasternal 2-chamber raw echocardiographic views routinely obtained during echocardiographic examination. By integrating various sources of information, the detection of PAH does not necessarily depend on the presence of TR. We submit that, by incorporating routine imaging data directly into the model the prediction, accuracy is increased compared with an approach relying only on selected numerical echocardiographic parameters and potentially opens our approach to general use, after appropriate validation. Our results compare favourably with those reported by Zhang et al.¹² who used 2D echocardiography from the apical 4-chamber view to detect PAH. The authors reported an area under curve for PAH detection of 0.85 against normal controls, albeit in the absence of a comparison group with non-PAH RV dilatation. The quality and clinical implications of the DL model presented with our novel data are illustrated by the ability to distinguish PAH patients from those with non-PAH RV dilatation. This distinction requires the model to incorporate morphological features beyond mere RV dilatation in the detection process. The plausibility of the detection algorithm is further illustrated by the gradient-weighted class activation mapping showing a coarse localization map that highlights the anatomically important regions for activation (Supplementary data online, Figure A).¹⁹ This highlights that RV and RA morphologic elements seem to represent relevant structures, which would coincide with intuition and clinical experience.

We do not advocate that the model, presented here, is currently suited to replace expert tertiary centre-based imaging and clinical expertise, nor do we suggest that the presented technology would outperform or obviate that need for such experts. Rather, we contend that this technology could be a useful adjunct in initial screening for PAH of patients in the community, where expertise in PAH might be less ubiquitous and merits further investment.

The prognostic power of the DL-based chamber quantification demonstrated here is remarkable and compares well with prognostic echocardiographic models that employ manually extracted echocardiographic measures from an expert PAH centre. We contend that this illustrates the ability of state-of-the-art machine learning models to assist patient assessment and guide therapy. While the accuracy of chamber quantification was not expected to be better compared with expert investigators, we were positively surprised by the accuracy of automated chamber quantification. The main clinical application of this technology is in improving the efficiency of the echocardiographic assessment and reporting of quantitative measures by non-expert investigators. Indeed, quantification of one echocardiographic frame only requires approximately 100 milliseconds on a portable commercial computer system. The algorithm also allows for time-efficient quantification of the entire cardiac cycle (not limited to systolic and diastolic measures). This is shown by the automatic quantification of LV eccentricity indices in the current study. This parameter has been shown to correlate well with invasive haemodynamics and outcome measures and can accurately identify individuals with clinically important PAH.²⁵ Perhaps, due to time constraints in routine clinical practice, the uptake of such a parameter has been slow. It is hoped that with the availability of more efficient DL-based systems, more parameters like this could be made available to frontline clinicians and assist in the management of PAH.

The aim of our study was to generate a tool that can expedite the diagnosis of PAH using DL techniques that allow automated primary echocardiographic detection of PAH. We did not intend to develop an entirely new echocardiographic prognostic model, nor do we suggest that a DL-based model should replace expert echocardiographic assessment of PAH patients in expert centres. Further investment in DL algorithms and acquisition of additional training data (including data from tertiary and non-tertiary centres) is required before the algorithms presented here may be deployed to routine care. However, by demonstrating that a DL-based model can provide sound prognostic data in PAH, further investment in algorithm improvement is called for, and should be in the realm of computer engineering and commercial corporations providing resources to this end, in collaboration with frontline clinicians and researchers.

Limitations

All PAH patients included were recruited from an expert setting and we cannot exclude the possibility that the predictive value of the DL models might be less accurate when applied to a community-derived PAH screening sample. We did not employ echocardiographic contrast to potentially increase the number of patients with visible and quantifiable tricuspid regurgitation jets and this may in part explain the considerable number of PAH patients without measurable tricuspid regurgitation in this study. Patients in the RV dilatation group and normal controls did not undergo invasive assessment and we cannot formally exclude the possibility that individual patients might be afflicted by pulmonary vascular disease in these groups. By design, the median follow-up of the study was short, corresponding to the time between the most recent echocardiographic investigation and death/last appointment. Therefore, the mortality rate described here is necessarily overestimated and not comparable to PAH studies investigating this issue specifically. This was an academic endeavour to develop models and assess the feasibility of a DL-based approach for the diagnosis and prognostication in PAH, with far reaching clinical potential. All models presented can be further refined by additional model training, or by incorporating more sophisticated model designs (e.g. recurrent networks); this should be attempted as part of external validation with appropriate industry support on image data from different centres, including non-tertiary cardiology units. Only all-cause mortality was used as a study endpoint. Further studies assessing the potential ability of DL-derived parameters to predict a composite of death and transplantation, cardiovascular death, and heart failure hospitalizations as well as to guide therapy in this setting are required.

Conclusions

Our data demonstrate for the first time the ability of machine learning algorithms, trained on existing echocardiographic datasets, to detect PAH, and distinguish it from RV dilatation not associated with pulmonary vascular disease, and normal controls. Automatic cardiac chamber segmentation and prognostication was also feasible with appropriate DL algorithms, and correlated with prognosis in this vulnerable PAH population. Therefore, DL methods may allow for improved screening and the development of tools for optimizing the management of PAH.

Supplementary Data

Supplementary material is available at European Heart Journal - Cardiovascular Imaging online.

Funding

None declared.

Data Availability

The data underlying this article cannot be shared publicly due to privacy concerns of individuals participating in the study. The anonymized imaging data will be shared on reasonable request to the corresponding author pending approval by the involved health care trusts.

References

Author notes