Comparison of the EuroSCORE and Cardiac Anesthesia Risk Evaluation (CARE) score for risk-adjusted mortality analysis in cardiac surgery^† Free

Abstract

OBJECTIVE

The European System for Cardiac Operative Risk Evaluation (EuroSCORE) and the Cardiac Anesthesia Risk Evaluation (CARE) score are risk indices designed in the mid-1990s to predict mortality after cardiac surgery. This study assesses their ability to provide risk-adjusted mortality in a contemporary cardiac surgical population.

METHODS

The mortality probability was estimated with the additive and logistic EuroSCORE, and CARE score, for 3818 patients undergoing cardiac surgery at one institution between 1 April 2006 and 31 March 2009. Model discrimination was obtained using the area under the receiver operating characteristics (ROC) curve and calibration using the appropriate chi-square goodness-of-fit test. Recalibration of risk models was obtained by logistic calibration, when needed. Calculation of risk-adjusted mortality was performed for the institution and eight surgeons, using each model before and when needed, after recalibration.

RESULTS

The area under the ROC curve is 0.72 (95% confidence interval (CI): 0.71–0.74) with the additive EuroSCORE, 0.84 (95% CI: 0.83–0.85) with the logistic EuroSCORE, and 0.79 (95% CI: 0.78–0.81) with the CARE score. The additive and logistic EuroSCORE have poor calibration, predicting a hospital mortality of 6.24% and 7.72%, respectively, versus an observed mortality of 3.25% (P < 0.001). Consequently, the risk-adjusted mortality obtained with those models is significantly underestimated for the institution and all surgeons. The CARE score has good calibration, predicting a mortality of 3.38% (P = 0.50). The hospital risk-adjusted mortality with the recalibrated additive and logistic EuroSCORE and CARE score is 3.24% (95% CI: 3.05–3.43%), 3.25% (95% CI: 3.05–3.44%), and 3.12% (95% CI: 2.94–3.34%), respectively. The individual surgeons' risk-adjusted mortality is similar with the recalibrated EuroSCORE models and CARE score, identifying two surgeons with higher rates than the hospital average mortality.

CONCLUSIONS

The original additive and logistic EuroSCORE models significantly overestimate the risk of mortality after cardiac surgery. However, after recalibration both models provide reliable risk-adjusted mortality results. Despite its lower discrimination as compared with the logistic EuroSCORE, the CARE score remains calibrated a decade after its development. It is as robust as the recalibrated additive and logistic EuroSCORE to perform risk-adjusted mortality analysis.

INTRODUCTION

Originally developed in the mid-1990s from a large multinational European population [1,2], the European System for Cardiac Operative Risk Evaluation (EuroSCORE) is a 17-risk-factor model predicting mortality either from a logistic regression equation (logistic EuroSCORE) [3] or an additive model (additive EuroSCORE) [2]. Tested and validated in numerous institutions throughout the world both as a logistic and an additive model, the EuroSCORE is one of the benchmark multifactorial risk indices with which other risk scores have been compared [4]. However, the large number of clinical variables and the regression equation to calculate risk may limit the routine use of the EuroSCORE by clinicians. The CARE score was specifically designed to overcome those difficulties. Developed in a single center 2 years after the EuroSCORE [5], the CARE score is a simple risk-ranking system, which resembles the well-known physical status classification of the American Society of Anesthesiologists, a model also familiar to many surgeons (Table 1). Despite its simplicity, the CARE score can predict perioperative mortality as well as the EuroSCORE [6] and other well-known multifactorial risk indices do [5].

The good predictive performance of the CARE score likely results from the sound clinical evidence supporting the definition of its risk categories. For example, the definition of the CARE 1 and CARE 5 categories comes from the fact that very-low- and very-high-risk cardiac surgical patients are easily identified by clinicians [7]. The subjective estimation of patients with intermediate risk is more inconsistent [7]. However, a few objective clinical variables significantly improve risk discrimination in those patients [8,9]. Thus, two broad, but objective groups of risk factors are used to define the intermediate risk levels of CARE 2–4: the surgical complexity and the presence of co-morbid conditions categorized as controlled or uncontrolled. The relative predictive importance of those covariables in the CARE score is consistent with the data from many logistic regression risk models, including the EuroSCORE [2,5]. Patients with controlled medical conditions (e.g., diabetes mellitus, cerebral or peripheral vascular disease, and so on) are at greater risk than patients without any co-morbidity but are at lower risk than patients with uncontrolled conditions (pulmonary edema, renal insufficiency, etc.); thus the rationale for the CARE 2 and CARE 3 categories. Uncontrolled co-morbid factors and complex/difficult procedures have comparable risk coefficients in many logistic regression models. Therefore, the same prognostic value is attributed to both groups of factors in the CARE score, explaining why one or the other group may define the CARE 3 category. The CARE 4 category is based on the fact that uncontrolled medical problems and complex surgery have an additive effect on risk. Finally, special consideration is given to emergency because emergencies and catastrophic states have the most important predictive value in many logistic regression risk models [2,5]. As a result, the CARE score has eight categories: CARE 1–5 for elective or urgent cases and CARE 3E, 4E, and 5E for emergency cases requiring immediate surgery.

One major purpose of predictive risk models in cardiac surgery is the performance of risk-adjusted outcome analysis. While the predictive characteristics of the EuroSCORE and CARE score are well known, their usefulness in conducting risk-adjusted outcome analysis has never been compared. This study is designed to assess the ability of the EuroSCORE and CARE score to provide risk-adjusted mortality in patients operated approximately a decade after the development of both models. The two primary objectives are: first, to assess the calibration of the EuroSCORE and CARE score in predicting perioperative mortality and second, to compare the overall and individual cardiac surgeons' risk-adjusted mortality at one institution, as calculated by the EuroSCORE and CARE score before and when needed, after recalibration of each model.

MATERIAL AND METHODS

Patients

With approval by the Human Research Ethics Board at the University of Ottawa Heart Institute, this study includes all consecutive surgical patients operated on between 1 April 2006 and 31 March 2009. The exclusion criteria are patients undergoing heart transplantation, insertion of ventricular assist device, pulmonary thromboendarterectomy, and trans-catheter aortic-valve implantation. Those procedures were excluded because some or all of them were not used for the development of the EuroSCORE or the CARE score.

Data collection

All data were extracted from the University of Ottawa Heart Institute perioperative database, which contains more than 400 variables related to the preoperative, intraoperative and postoperative condition of each patient. All data are collected in a prospective manner. The preoperative and intra-operative data are first entered by the attending anesthesiologist who also attributes a CARE score to their patients at the time of surgery. The completeness and accuracy of the preoperative and intra-operative data collection are verified daily by two research assistants who also collect the postoperative data. Patients undergoing more than one cardiac surgical procedure during the same hospital admission are counted as single cases, the first surgery being used for analysis. The reported mortality represents any intra- or postoperative in-hospital death during a same admission, regardless of the cause or patient length of stay in hospital.

Data analysis

The association between the EuroSCORE risk factors and mortality was determined by univariate analysis, using a chi-square test or a Fisher exact test when appropriate. Three probabilities of death were estimated for each patient. The first one was derived from the additive EuroSCORE, which is a three-category model attributing a fixed probability of death for each risk category [2]. The second probability of death was calculated using the original equation of the logistic EuroSCORE: probability of death = e^Z/(1 + e^Z) where Z = B₀ + B₁X₁ + …+ B₁₇X₁₇, B₀ is the regression intercept, and B₁–B₁₇, the regression coefficients for each (X_n) risk factor [3]. Finally, death was estimated using the CARE score, which has a fixed probability of death for each of its eight risk categories [5]. Discrimination of the three risk classifications for mortality was assessed through receiver operating characteristics (ROC) curves. Pair-wise comparisons of the area under the curves were made using the methods of Hanley and MacNeil [10]. Calibration, or comparison between the predicted and observed mortality, was assessed for the additive EuroSCORE and CARE score using the Pearson chi-square goodness-of-fit test [11]. As the logistic EuroSCORE has no fixed categories, its calibration was assessed using the Hosmer—Lemeshow chi-square method [12].

Recalibration of the EuroSCORE models was performed by shifting the regression line relating the observed mortality with the total score (points) of the additive EuroSCORE and with the predicted log odds of the logistic EuroSCORE. This approach has been recommended for recalibration of logistic regression models with relatively small samples [13]. This calibration process consists in updating the regression intercept B₀ and the overall calibration slope B_overall by fitting a logistic regression model to the study sample. The logistic regression model is obtained by using the total score as the only covariable in the recalibrated additive EuroSCORE and by using the sum of the B_nX_n as the only covariable in the recalibrated logistic EuroSCORE. With this approach, the recalibrated logistic EuroSCORE model maintains all its original regression coefficients and their relative predictive ratios. Similarly, the recalibrated additive EuroSCORE has the same scoring system, with each point having a readjusted predictive value. For both recalibrated EuroSCORE models, fitting between the observed and predicted mortality was assessed in the entire population by the Hosmer—Lemeshow chi-square method. To ascertain the stability of the equations to predict mortality with both models, the population was split randomly into reference and validation samples of equal size 10 times. For both recalibrated models, the means of the regression intercepts B₀ and variable regression coefficients were compared between the 10 reference and validation samples, using paired samples' t-test.

For each risk model, risk-adjusted mortality was calculated for the hospital and eight individual surgeons who performed all the surgical cases during the study period. The equation to calculate risk-adjusted mortality is: ((observed mortality/predicted mortality) × hospital average mortality). The unadjusted predictive weight of the EuroSCORE risk factors is presented as odds ratio (OR). The predicted and risk-adjusted mortality values are given in percentages with their 95% confidence interval (CI). Results from the random split data analyses are given as means of the regression intercepts B₀ and variable regression coefficients, with their standard error of the means. The P value given with the predicted and risk-adjusted mortality rates is for comparisons made with the hospital average mortality, using a chi- square test. For all comparisons, P value <0.05 was considered statistically significant.

RESULTS

A total of 3818 patients were included in the study. The overall in-hospital mortality rate was 3.25% (124 patients). Table 2 summarizes the patient characteristics as defined by the EuroSCORE risk factors and their unadjusted association with mortality.

The area under the ROC curve for mortality prediction is 0.72 (95% CI: 0.71–0.74) with the additive EuroSCORE, 0.84 (95% CI: 0.83–0.85) with the logistic EuroSCORE and 0.79 (95% CI: 0.78–0.81) with the CARE score. There is significant difference in discrimination between the additive and logistic EuroSCORE (P < 0.001) and between the additive EuroSCORE and the CARE score (P = 0.015). The difference does not reach significance between the logistic EuroSCORE and the CARE score (P = 0.057).

The additive and logistic EuroSCOREs have poor calibration and overestimate mortality at each of their respective risk levels (Tables 3 and 4). The average hospital mortality predicted by the additive and logistic EuroSCORE is 6.24% (95% CI: 6.10–6.39) and 7.72% (95% CI: 7.39–8.06), respectively. Both values are significantly higher than the observed mortality of 3.25% (P < 0.001). The CARE score has good calibration (Table 5) and predicts an average hospital mortality of 3.38% (95% CI: 3.16–3.59; P = 0.75).

The mortality predicted by the recalibrated EuroSCORE is calculated with the following equations:

predicted mortality = e^{[−5.975+(0.342×total score)]}/1 + e^{[−5.975+(0.342×total score)]} with the additive model;

predicted mortality = e^{[−5.936+(1.041×ΣB_nX_n)]}/1 + e^{[−5.936+(1.041×ΣB_nX_n)]} with the logistic model

Those equations provide good fit between the predicted and the observed mortality for the recalibrated additive (chi-square = 7.57; df = 9; P = 0.58) and logistic (chi-square = 5.80; df = 9; P = 0.76) EuroSCORE (Fig. 1). The stability of those equations is confirmed by the random split data analyses. The mean regression intercept B₀ of the reference and validation samples is −5.902 ± 0.055 versus −6.083 ± 0.059 (P = 0.148) for the recalibrated additive EuroSCORE, respectively, and −5.872 ± 0.054 versus −6.033 ± 0.049 (P = 0.193) for the recalibrated logistic EuroSCORE, respectively. The mean variable regression coefficient of the reference and validation samples is 0.335 ± 0.004 versus 0.352 ± 0.005 (P = 0.101) for the additive model, respectively, and 1.023 ± 0.015 versus 1.067 ± 0.017 (P = 0.202) for the logistic model, respectively.

The areas under the ROC curve for mortality prediction are 0.84 (95% CI: 0.80–0.87) and 0.84 (95% CI: 0.81–0.87) for the recalibrated additive and logistic EuroSCORE, respectively. The overall predicted mortality is 3.24% (95% CI: 3.05–3.43%: P = 0.99) and 3.25% (95% CI: 3.05–3.44: P = 1.0) with the recalibrated additive and logistic EuroSCORE, respectively.

The hospital risk-adjusted mortality is 1.69% (95% CI: 1.65–1.73) by calculation with the original additive EuroSCORE and 1.37% (95% CI: 1.31–1.43) with the logistic EuroSCORE. Both values are significantly lower (P < 0.001) than the observed hospital average mortality (3.25%). Using these models, the individual surgeons' risk-adjusted mortality and 95% CI are also much lower than the observed hospital average mortality (Fig. 2). The hospital risk-adjusted mortality obtained with the recalibrated additive and logistic EuroSCORE and CARE score is 3.26% (95% CI: 3.07–3.46; P = 0.99), 3.25% (95% CI: 3.05–3.44; P = 1.0), and 3.12% (95% CI: 2.94–3.34%; P = 0.744), respectively. Those three values are comparable and not significantly different from the observed hospital average mortality. The analyses of individual surgeons' risk-adjusted mortality with the two recalibrated EuroSCORE models and the CARE score provide profiles that are comparable but quite different from those with the original EuroSCORE models, identifying two surgeons with values significantly higher than the observed hospital average mortality (Fig. 3).

DISCUSSION

We found that the additive and logistic EuroSCORE significantly overestimate the risk of mortality after cardiac surgery in our center. Consequently, the EuroSCORE underestimates risk-adjusted mortality. However, after recalibration both models provide reliable risk-adjusted mortality results. We also found that, a decade after its development, the CARE score has maintained its calibration for mortality prediction in our institution. As a result, it provides risk-adjusted mortality rates for the hospital and individual surgeons similar to those obtained with recalibrated EuroSCORE models. Those findings are important for clinicians because they highlight the fact that mortality prediction and risk-adjusted outcome analysis in cardiac surgery can be reliably performed without risk models based on complex mathematical equations.

Loss of calibration with the additive and logistic EuroSCORE has been observed by many investigators throughout the world since 2003 [4,14–17]. One possible reason for this frequent finding is that the EuroSCORE was developed from patients undergoing surgery more than 15 years ago. Surgical techniques and perioperative care have evolved considerably since then. This has most likely resulted in the decreased mortality rate observed worldwide, despite the fact that older patients with more co-morbid conditions undergo cardiac surgery. Another explanation for the poor EuroSCORE calibration may be inherent structural characteristics limiting the reproducibility of its predictions. The most obvious structural limitation is the large number of risk factors in the model, many of them being highly correlated. For example, emergency surgery, critical preoperative condition, recent myocardial infarction, unstable angina requiring intravenous nitroglycerin, poor left-ventricular function, pulmonary hypertension, and post-infarct septal defect may all have the same clinical variable as a denominator. As demonstrated in a previous study, the large number of variables in the EuroSCORE is associated with significant mortality over-estimation in patients at intermediate risk [15]. The effect is less important in low- or high-risk patients. It was also found that mortality prediction with only five of the EuroSCORE variables (age, left-ventricular ejection fraction, serum creatinine, emergency operation, and non-isolated coronary operation) provides the same level of discrimination and better calibration than with the 17-risk factor model [15]. Those results are supported by other studies which also showed that risk models with a limited number of variables provide accurate and reproducible risk predictions in cardiac surgery [8,9,18]. The good predictive performance of the CARE score in this study represents additional evidence supporting those findings.

The numerous reports of poorly calibrated predictions with the EuroSCORE have likely prompted the revision of the model by the EuroSCORE group of investigators. The revised version is not available yet (http://www.euroscore.org/). Until completion and publication of the updated EuroSCORE, recalibration of its predictions is necessary to perform risk-adjusted outcome analysis in most cardiac surgical centers. The most rigorous approach to recalibrate a risk model is through re-estimation of the regression coefficient of all its variables. In relatively small populations, it is difficult with that method to obtain a stable recalibrated model with 17 risk factors. This is particularly true when mortality and the prevalence of certain variables (e.g., post-infarct septal rupture and active bacterial endocarditis) are as low as in this study. This is why the EuroSCORE was recalibrated in this study using the logistic calibration method [13]. Both recalibrated models were stable, had good discrimination, and provided a good fit between the observed and predicted mortality, making them appropriate to assess our institution and individual surgeons' risk-adjusted mortality.

There are at least three possible reasons why the CARE score had good calibration in this study. The first one is the fact that the analysis was performed in the institution where it was developed. This likely limited the heterogeneity between the studied population and the original case mix used to derive the CARE score predictions. The second reason may be that the CARE score uses only a small number of robust variables, which make the predictive model more stable. The final reason may simply be a random effect. Like all predictive models, the CARE score will overestimate risk as the patient population, the surgical techniques and the perioperative care continue to evolve. In fact, a close look at the mortality among the very-high-risk patients (Table 5) suggests a trend for risk overestimation with the CARE score in that population. Similar results are seen with most risk models in cardiac surgery [4,17]. Thus, although not needed in this study, recalibration will eventually be required.

Like many predictive risk models, the CARE score was developed in the 1990s when reporting of risk-adjusted cardiac surgical outcomes became a medical and public topic of interest [19,20]. In that context, the CARE score may not appear as the ideal risk model because it allows rater's subjectivity, which can be used to overrate risk and calculate lower risk-adjusted mortality. However, the CARE score was never meant to be used by professional or government bodies to provide outcome reports. Based on the fact that a few clinical variables and clinical judgment alone provide a large amount of prognostic information, it was developed as a system to rank patient risk much as the New York Heart Association classification is used to grade the severity of patient symptoms. In view of its good predictive performance, it was also proposed as a simple means for clinicians to periodically measure their own risk-adjusted results to monitor their practice [5]. This study shows that the CARE score can achieve that objective. It provides institutional and individual surgeons risk-adjusted mortality profiles comparable to those obtained with the recalibrated additive and logistic EuroSCORE.

The fact that the CARE score performs as well as the recalibrated EuroSCORE highlights a few aspects about risk prediction that clinicians must consider when using risk models to assess their practice. First, none of those models have the accuracy of a diagnostic test. The discrimination provided by those models (0.75–0.85) is a borderline signal detection that is actually not better than that obtained by weather forecasting for rain (0.75–0.90) [21]. Second, logistic regression models are useful in identifying clinical variables associated with poor outcome. They also provide the relative contribution of those variables to poor outcome. However, predictions derived from those models are barely more accurate than those obtained through clinical judgment [4,7]. This is probably because outcomes depend on numerous biological variables and human factors that cannot be computed by mathematical equations. Consequently, risk-adjusted results based on any of those models must always be interpreted with caution.

Because this is a single-center study, we are unable to determine if our results can be generalized. This issue can only be resolved by repeating similar analyses in other institutions. However, most of our findings relating to the predictive performance of the EuroSCORE and CARE score have previously been observed by other investigators. Therefore, it is quite plausible that our results can be replicated in other institutions. This may require recalibration of the CARE score. However, the major conclusion would likely remain the same: risk-adjusted mortality analysis can be reliably performed at a local level with a risk model as simple as the CARE score.

In conclusion, this study shows that the original additive and logistic EuroSCOREs significantly overestimate risk in patients who had cardiac surgery in our institution between 2006 and 2009. However, after recalibration they are appropriate for risk-adjusted mortality analysis. A decade after its development, the CARE score maintains good calibration for perioperative mortality prediction in the institution where it was originally tested. Like all other risk models in cardiac surgery, the CARE score might lose its calibration over time. However, it is currently as robust as the recalibrated additive and logistic EuroSCORE to perform risk-adjusted mortality analysis in our institution.

Funding

This study is supported by The Research Funds of the Cardiac Surgical Intensive Care Unit and the Cardiac Anesthesia Division of the University of Ottawa Heart Institute, Ottawa, Ontario, Canada.

Conflict of interest: none declared.

ACKNOWLEDGMENTS

The authors thank Dr Lily Tong and Dr Sandra Vranjes for their professional work in collecting and managing the data for this study.

REFERENCES

Author notes