Validity of the Swedish Cardiac Surgery Registry Free

Abstract

OBJECTIVES

Our goal was to validate the Swedish Cardiac Surgery Registry by reviewing the reported cardiac operations to assess the completeness and quality of the registered data and the EuroSCORE II variables.

METHODS

A total of 5837 cardiac operations were reported to the Swedish Cardiac Surgery Registry in Sweden during 2015. A randomly selected sample of 753 patient records (13%) was scrutinized by 3 surgeons at all 8 units in Sweden performing open cardiac surgery in adults.

RESULTS

Coverage was excellent with 99% [95% confidence interval (CI) 98–99%] of the performed procedures found in the registry. Reported waiting times for surgery were correct in 78% (95% CI 76–79%) of the cases. The main procedural code was correctly reported in 96% (95% CI 95–97%) of the cases. The correlation between reported and monitored logistic EuroSCORE II had a coefficient of 0.79 (95% CI 0.76–0.82), and the median difference in EuroSCORE II was 0% (interquartile range −0.4% to 0.4%). The majority of EuroSCORE II variables had good agreement and coherence; however, New York Heart Association functional class, preoperative renal dysfunction, left ventricular ejection fraction, Canadian Cardiovascular Society Class IV angina and poor mobility were less robust. Postoperative complications were rare and in general had a high degree of completeness and agreement.

CONCLUSIONS

The reliability of the variables in the national Swedish Cardiac Surgery Registry was excellent. Thus, the registry is a valuable source of data for quality studies and research. Some EuroSCORE II variables require improved and stricter definitions to obtain uniform reporting and high validity.

INTRODUCTION

Presently available national health databases and registries have enabled rapid access to a large amount of information about thousands of individuals that can be used for data analyses [1]. High validity of registry data is a prerequisite because the data are increasingly used in scientific analyses and as references in reports of clinical outcome [1]. Routinely collected clinical data may be questionable to refer to if they are not validated [1, 2]. Large registries and databases containing assessments of diagnoses, diagnostic techniques and procedures and treatments have been validated [1–5]. The Swedish Cardiac Surgery Registry was formed in 1992 and comprises all cardiac operations performed on adults in Sweden. In December 2009, the registry was merged with the national health quality database, the Swedish Web-system for Enhancement and Development of Evidence-based Care in Heart Disease Evaluated According to Recommended Therapies (SWEDEHEART) registry [6]. There is continuous monitoring of most of the registries within SWEDEHEART, but the data quality of only 1 diagnosis has been published [7]. Systematic validation and adjudication of the entire Swedish Cardiac Surgery Registry have not been conducted previously. EuroSCORE II is used in Sweden to assess the risk of a cardiac operation [8].

To assess completeness and quality of data and particularly EuroSCORE II factors reported to the Swedish Cardiac Surgery Registry, a randomly selected sample of cardiac operations performed during 2015 was validated for several key variables.

METHODS

All 8 units in Sweden performing open cardiac surgery in adult patients report information such as date of surgery, patient characteristics, EuroSCORE II factors, type of and details of the operation, postoperative complications and medications to SWEDEHEART. The central server is managed by the Uppsala Clinical Research Center (UCR) located in Uppsala, Sweden. The Swedish Cardiac Surgery Registry is sponsored by the Swedish Health Authorities and is independent of commercial funding. Annual reports of aggregated anonymous results are publicly available online (www.swedeheart.se).

Data are not reported uniformly. Five units export encrypted data from local databases, 2 units report directly online through an Internet interface and 1 unit uses combined Internet and standardized registration forms. The study was approved by the Regional Ethical Review Board in Stockholm (2016/2331-31).

Patient selection

All patients undergoing a cardiac operation in Sweden during 2015 were potential subjects for validation. Each unit was visited for 2 days during 2016 by a head supervisor and a surgeon from another unit in a rotating schedule. The head supervisor visited all units to ensure that identical criteria for the validation were followed. The validation was conducted by 3 experienced cardiac surgeons including the local representative who was familiar with the hospitals’ electronic patient chart systems. To ensure that a random selection of cardiac operations performed during 2015 was validated, all operations performed during a specific day were selected, starting with the 10th of each month regardless of holidays and weekends. When all months of the year were completed, the 11th of each month was also validated and so forth. Because validation was conducted for 2 days, approximately the same number of patients was reviewed at each unit. However, the proportion of patients validated at each unit varied due to differences in case load and varying possibilities to find variables in the electronic systems.

Variables

The patient records were reviewed for days waiting for surgery, date of the operation, surgical procedure codes, EuroSCORE II factors and postoperative complications. The EuroSCORE II includes 18 patient, cardiac and operation-related factors including age, sex, preoperative renal dysfunction (calculated creatinine clearance based on age, sex, weight and preoperative creatinine) [9], extracardiac arteriopathy (ECA), poor mobility due to musculoskeletal or neurological dysfunction, previous cardiac surgery, chronic pulmonary disease (CPD), active infectious endocarditis, critical preoperative state, insulin-dependent diabetes mellitus, New York Heart Association (NYHA) class, Canadian Cardiovascular Society (CCS) Class IV angina (i.e. angina at rest), left ventricular ejection fraction, myocardial infarction within 90 days, systolic pulmonary artery pressure (SPAP), urgency of the procedure (i.e. elective, urgent, emergency or salvage), weight of the intervention (i.e. isolated coronary artery bypass grafting, isolated non-coronary artery bypass grafting, 2 cardiac procedures or 3 procedures) and surgery on the thoracic aorta. Details regarding definitions of variables and calculation of EuroSCORE II are described elsewhere [8]. The local EuroSCORE II classification was done by the performing surgeon at 5 units and by another physician at 3 units.

The variables selected for validation included the following complications: reoperation for bleeding ≤24 h of the index operation, reoperation for deep sternal wound infection or sternal dehiscence before discharge (defined as rewiring), reoperation for coronary graft failure or for heart valve dysfunction before discharge, postoperative stroke defined as circulatory-related central neurological deficit lasting ≥72 h, need for postoperative renal replacement therapy and postoperative antibiotic treatment other than routinely given surgical prophylaxis.

Data were entered directly online during the monitoring process into a mirror database identical to the original SWEDEHEART database. The patients were identified by the unique personal identity number assigned to all Swedish permanent residents [10]. This procedure enabled individual-level matching to originally reported data. Data acquired during the monitoring process for each patient were used as a reference and compared with the reported data in the SWEDEHEART database. Based on the number of patients found in the originally reported database, the coverage of the registry as well as the validity of each of the variables could be assed. The validation process thus comprised both an inquiry of the completeness of data and a detailed adjudication to ensure that definitions and classifications were correct.

Statistical analysis

Waiting time for surgery was considered correct if the difference between the number of days obtained during the monitoring process and the reported number of days was ≤2 days. The operation codes were considered correct if all operation codes entered during the monitoring could be found in the originally reported data. Differences in EuroSCORE II were calculated by subtracting the value obtained during monitoring from the originally reported value for each patient. In addition, the correlation of agreement between the monitored and reported waiting times and EuroSCORE II was calculated using Pearson’s method.

The total EuroSCORE II agreement was calculated as the percentages of the reported observations that were higher and lower than the monitored data that were used as references. In addition, coherence between the reported and monitored recordings was measured using Kappa statistics for nominal variables and Kendall’s W statistics for ordinal variables. Thus, the validity of each variable was assessed by both agreement and coherence. A Kappa/Kendall statistic of ≤0 indicates coherence no better than chance, and the traditional interpretation is 0–0.20 as slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1 as excellent coherence [11]. Statistical tests are not shown for Kappa/Kendall statistics, because very low values are required to reject the null hypothesis.

Data are presented with a 95% confidence interval (CI), except for differences in EuroSCORE II values with skew distribution, which are expressed as the median with the interquartile range. The statistical program R was used (v. 3.3.0) [12].

RESULTS

A total of 5837 cardiac operations were reported to the Swedish Cardiac Surgery Registry in 2015. The validation included 753 procedures (13%). This random sample comprised cases similar to the cases not validated (n = 5094) regarding patient characteristics and complications (Fig. 1A and B, Tables 1 and 2). Between 81 and 106 patients were reviewed at each unit, corresponding to 8–32% of cases performed during 2015 (Fig. 2A).

Among the validated procedures, 743 were identified in the originally reported database. Thus, the coverage of the registry was estimated to be 99% (95% CI 98–99%). Reporting to SWEDEHEART had failed in 10 cases. Two cases were not identified because of temporary personal identity numbers, 2 had errors in the data export format and in 6 instances reporting was overlooked.

Waiting time

During validation, the date of acceptance for surgery was identified for 99.5% of cases (n = 749). Reported waiting times for surgery were correct in 79% (95% CI 78–81%) of the procedures. The correlation between reported and monitored waiting times was good with a coefficient of 0.76 (95% CI 0.73–0.79). In case of disagreement, the reported waiting times were generally shorter than waiting times calculated at the validation (Supplementary Material, Table S1).

Procedure codes

The main procedural code was correctly reported in 96% (95% CI 95–97%) of the cases. All procedure codes associated with the reported operation were correct for 85% (95% CI 84–86%) of the procedures.

EuroSCORE II

The median difference between reported and monitored EuroSCORE II values was generally small (Fig. 2B). However, clinic C had reported lower EuroSCORE II values than those recorded at the validation, with a median difference of −0.4% (interquartile range −2.0% to 0%). The median difference for the entire Swedish Cardiac Surgery Registry was 0% (interquartile range −0.4 to 0.4%).

There was good correlation between reported and monitored EuroSCORE II values with a coefficient of 0.79 (95% CI 0.76–0.82). Poorer correlation was mainly observed for higher EuroSCORE II values, similarly distributed among all units (Fig. 3).

EuroSCORE II variables

Most EuroSCORE II variables had good agreement and substantial coherence between reported and monitored data (Fig. 4A–C). The factors were almost completely reported except for NYHA class and preoperative renal dysfunction with a coverage of 98% (95% CI 97–98%) and 94% (95% CI 94–95%), respectively (Fig. 4B). Some of the EuroSCORE factors displayed an agreement of <90%, i.e. NYHA class, preoperative renal dysfunction, left ventricular ejection fraction, recent myocardial infarction, SPAP and weight of the procedure (Figs 4A and 5). Further, some of the EuroSCORE II variables had a less than moderate coherence despite good total agreement, i.e. CCS IV and poor mobility (Fig. 4C).

The variable with poorest agreement was NYHA class. Reported and monitored data differed in more than half of the patients. The class recorded at the monitoring process was both lower and higher than that originally reported (Fig. 4A). However, a bias towards underestimation was more frequent. Other variables that also demonstrated a bias towards underestimation in the original data were preoperative renal dysfunction, active infectious endocarditis and weight of procedure. On the other hand, some of the variables were systematically overestimated, i.e. CCS Class IV angina, ECA, CPD, poor mobility, critical state, depressed left ventricular ejection fraction, recent myocardial infarction, SPAP and aortic surgery.

Postoperative complications

There was in general good agreement between the reported and monitored data regarding postoperative complications (Fig. 6A). Antibiotic treatment in addition to surgical prophylaxis was an exception, with 9.3% (95% CI 7.2–11%) missed entries in the original data. Reoperation for coronary occlusion and need for postoperative dialysis were also underestimated whereas postoperative stroke was overestimated. The coverage for monitored postoperative complications was complete, except for reoperation for coronary occlusion and valve dysfunction (Fig. 6B). However, reoperation for coronary occlusion and for valve dysfunction only involved 2 patients. Reoperation for bleeding demonstrated an excellent coherence, and sternal wound infection and postoperative dialysis both displayed a substantial coherence (Fig. 6C). Despite good agreement, the other postoperative complications (i.e. stroke, reoperation for coronary artery dysfunction and antibiotic treatment) displayed only a moderate coherence, and reoperation for valve dysfunction displayed coherence no better than chance.

DISCUSSION

This study of data reported to SWEDEHEART in 2015 showed, in agreement with other validation reports, high degrees of completeness and validity [3, 4, 13]. The Swedish Cardiac Surgery Registry had almost complete coverage of all open cardiac surgical procedures conducted in Sweden. The completeness of data was generally high regarding waiting time, procedure codes, EuroSCORE II variables and the postoperative complications. Furthermore, there was overall good agreement and coherence between reported and validated data.

Previous validations of variables in local databases have been conducted, but, to our knowledge, this is the first monitoring and adjudication process conducted in a major national cardiac surgery registry. The fact that the register had almost complete coverage and reliable validated data and can be cross-linked to other national health data registries, makes it a unique and invaluable source for quality improvements and research.

Still, there is room for improvement in data quality and completeness, which must be considered in studies and quality reports based on the registry. For instance, there were marked differences between originally reported and monitored observations for some EuroSCORE II variables (e.g. NYHA class, CCS Class IV, ECA, CPD, poor mobility, critical state) and postoperative antibiotic treatment. Accuracy regarding waiting times and procedure codes can also be improved.

Mechanisms behind errors

The errors found in the Swedish Cardiac Surgery Registry were perceivably caused by one or several of the following reasons: variable assessment or definition, negligence, technical issues and errors in the monitoring process.

As mentioned, NYHA class is a typical example of a variable that proved hard to assess. Firstly, it is subject to clinical judgement. Secondly, despite being originally a classification for heart failure, NYHA class was often, improperly, used for grading exertional limitations due to angina pectoris or musculoskeletal disease. However, the distinction between angina (CCS class) and heart failure (NYHA class) is highly theoretical and in practice impossible to determine in patients with the common combination of heart failure and coronary disease. Thirdly, in rare conditions like circulatory shock due to acute aortic dissection, the NYHA class cannot, in our opinion, be defined in a relevant way. There is in fact no consistent method to clinically assess heart failure that results in great interoperator variability, as supported by a previous study that demonstrated that the agreement between 2 experts grading NYHA class was only about 55% [14]. For this reason, each variable was reviewed by 3 surgeons during the monitoring to reach a consensus as to how the variable should be classified, which is a major strength of this study. However, despite low agreement on the exact NYHA class, the coherence was substantial, which suggested that in cases of disagreement between reported and validated data the difference was generally not greater than 1 NYHA class. Interestingly, despite the repeatedly proven unreliable nature of NYHA class, it is still used and reported in almost all cardiac-related publications [14]. We suggest it is time to attenuate the importance of NYHA class and to strive to use a more reproducible assessment of heart failure, such as pro-B-type natriuretic peptide levels [15, 16].

A definition can be a problem either because it is too vague or because it is misunderstood. An example of the former is poor mobility due to neurological or musculoskeletal disease. This definition is highly subjective, which is illustrated by poor coherence according to the Kappa statistics. An example of the latter is CPD, which is defined in EuroSCORE as ‘long term use of bronchodilators or steroids for lung disease’, and is thus not defined as impaired lung function alone. Despite a clear definition, this variable is often misused for patients with impaired lung function without medication or with restrictive lung diseases.

Examples of negligence are the cases with missing procedure codes and false negative entries in the variables ‘Reoperation’ and ‘Endocarditis’, both of which have a profound impact on patient care and should not be missed during registration.

Technical issues can arise through mistakes made when data are entered in the online module or during exportation of data from local registries. An example of the latter is the missing data in the variables ‘Coronary occlusion’ and ‘Valve dysfunction’ where exportation from 1 unit failed during 2015.

Finally, error can be caused by wrong entries during the monitoring process as discussed under limitations.

Methods to improve data quality

Variables that require a subjective assessment should be avoided and, if possible, should be replaced or strengthened with a more objective variable. On the other hand, requiring radiological/pathological evidence of a clinically suspected stroke instead of only a clinical judgement might, for instance, lead to underreporting of the event and improperly punish units with the high ambition of radiological examination based on broad indications.

It is also of paramount importance to educate the users in how the variables are defined, especially because it is not uncommon that personnel with limited experience in cardiac surgery complete parts of the registry. The importance of striving to construct variables with intuitive, clear definitions that are unlikely to be misunderstood cannot be overestimated.

Further, variables with a very low incidence ought to be kept to a minimum, because it is hard to achieve good validity with them. Reoperation for valve dysfunction, for instance, has a very low incidence (0.2%); thus, it was easy to reach a high level of agreement whereas the coherence was no better than chance, as indicated by the Kappa statistic. The reason for this result is that such uncommon complications are easily overlooked during registration, and it takes a few false negative or positive entries for the entire variable to be completely invalid. On the other hand, clinically important events in cardiac surgery are indeed fairly uncommon (e.g. stroke, valve repair failure), which makes this task challenging.

Errors due to negligence are perhaps the hardest to avoid. However, communicating the importance of a quality registry with correct data to the users and regularly monitoring the quality of the data will hopefully decrease these types of errors.

Limitations

An important limitation of this study is that the monitoring is retrospective based on patient charts. Thus, undocumented information regarding the patients’ history, status or investigational findings could cause entries during the validation to be erroneous and thereby wrongfully falsify the originally reported data. Some variables are particularly vulnerable in this regard, e.g. NYHA class, ECA, poor mobility and SPAP. Further, calculation of creatinine clearance in the EuroSCORE II algorithm requires the patient’s preoperative weight, which is often not obtained for patients operated on emergently, which might explain why some of the data related to this variable are missing.

EuroSCORE is a preoperative risk assessment score that is meant to be completed before surgery and is done so at the majority of the hospitals validated [8]. Since this is a retrospective validation, EuroSCORE values had to be filled in after surgery. Sometimes, the planned procedure is not the actual one carried out, which can account for some of the discrepancy found between the original data and validation.

Additionally, the Kappa statistic should be interpreted with caution because it has its limitations, i.e. the reliability of a variable should not be dismissed based solely on a low Kappa value. Instead, the Kappa value should be interpreted in its context with total agreement and false negative/positive entries [17]. In situations where an outcome is uncommon (e.g. sternal wound infection, stroke), it is difficult to get a high Kappa value due to so-called symmetrical imbalance, a phenomenon known as the Kappa paradox.

CONCLUSIONS

The reliability of the variables in the national Swedish Cardiac Surgery Registry was excellent. This outcome makes the registry a unique and invaluable source for quality improvements and research in cardiac surgery. An important finding in this study was that a monitoring process could identify weaknesses related to data quality and reporting. Work will continue to improve the data quality and reliability of the registry based on the lessons learned from this validation.

SUPPLEMENTARY MATERIAL

Supplementary material is available at ICVTS online.

ACKNOWLEDGMENTS

The authors are grateful for the kind cooperation and expert assistance of Christina Bellman at Uppsala Clinical Research Center (UCR), Uppsala, Sweden.

Conflict of interest: none declared.

REFERENCES