The use of an electronic health record system reduces errors in the National Hip Fracture Database Free

Abstract

Aim

to compare the validity of data submitted from a UK level 1 trauma centre to the National Hip Fracture Database (NHFD) before and after the introduction of an electronic health record system (EHRS).

Patients and methods

a total of 3224 records were reviewed from July 2009 to July 2017. 2,133 were submitted between July 2009 and October 2014 and 1,091 between October 2014 and July 2017, representing data submitted before and after the introduction of the EHRS, respectively. Data submitted to the NHFD were scrutinised against locally held data.

Results

use of an EHRS was associated with significant reductions in NHFD errors. The operation coding error rate fell significantly from 23.2% (494/2133) to 7.6% (83/1091); P < 0.001. Prior to EHRS introduction, of the 109 deaths recorded in the NHFD, 64 (59%) were incorrect. In the EHRS dataset, all the 112 recorded deaths were correct (P < 0.001). There was no significant difference in the error rate for fracture coding. In the EHRS dataset, after controlling for sample month, entries utilising an operation note template with mandatory fields relevant to NHFD data were more likely to be error free than those not using the template (OR 2.69; 95% CI 1.92–3.78).

Conclusion

this study highlights a potential benefit of EHR systems, which offer automated data collection for auditing purposes. However, errors in data submitted to the NHFD remain, particularly in cases where an NHFD-specific operation note template is not used. Clinician engagement with new technologies is vital to avoid human error and ensure database integrity.

Key points

• Large medical databases are error-prone, limiting their use as a research and audit tools.

• Our findings show the use of an EHRS reduces errors in the NHFD.

• As EHRS use becomes widespread, databases should become increasingly reliable, empowering researchers.

• However, clinician engagement with this new technology remains a limiting factor in ensuring database integrity.

Introduction

Fragility fractures present a key challenge for healthcare services internationally. In the UK, they account for ~£2 billion of the healthcare budget, with hip fractures representing 95% of this expenditure [1, 2]. In 2016, 65,645 patients presented to hospitals in England, Wales and Northern Ireland with hip fractures, and the annual number of admissions throughout the UK is expected to exceed 100,000 by 2020 [3]. Outcomes are poor, with 12-month mortality as high as 37% [1]. Evidently, these patients represent a clinical and economic challenge which will have an increasing impact on healthcare policy in the coming decades. In response to this challenge, the British Orthopaedic Association (BOA) and the British Geriatrics Society (BGS) collaborated to produce both the National Hip Fracture Database (NHFD) and the 2007 Blue Book for the care of patients with fragility fractures [1, 4]. These collaborative ventures aimed to set nationwide standards of care for patients sustaining fragility fractures. The NHFD is a national clinical audit that allows care at individual hospitals to be measured against evidence-based standards laid out in the Blue Book [4]. Data on each patient are entered into the NHFD via the project website. It includes demographics, time of admission, time of orthogeriatric review, time of surgery, fracture pattern, operation type and discharge information. Data entry is performed by designated health professionals at each hospital [4]. The introduction of the NHFD has been associated with improvements in patient outcome [5].

In addition to improving patient care, the NHFD serves as a large-scale research tool that can facilitate evidence-based policy change in hip fracture treatment [6]. However, some studies have highlighted errors in hip fracture databases that raise concerns over their utility in this regard [7, 8]. One potential solution to this issue is the use of an electronic health record system (EHRS). These allow real-time, standardised recording of audit data through templates completed by the clinician at the time of treatment [9]. To our knowledge, no study has examined the effect of an EHRS on errors within a national database.

We aimed to investigate the effect of an EHRS on the validity of data submitted from a UK level 1 major trauma centre to the NHFD. We compared our centre’s NHFD data with locally held data from various sources to audit its integrity before and after the introduction of an EHRS.

Materials and methods

Patient data

Ethical approval was obtained from the local ethics body (clinical project ID number 7023).

Data submitted to the NHFD from our level 1 major trauma centre between July 2009 and July 2017 were included, totalling 3,251 episodes. 2,156 of these occurred between July 2009 and October 2014 and 1,095 between October 2014 and July 2017, representing data submitted before and after the introduction of the EHRS. All patient-identifiable data were removed from the dataset prior to analysis. All analyses were performed at our centre. The NHFD data were compared to local data held on hospital administrative systems, Picture Archiving and Communication System (PACS) and patient notes (paper or electronic). For each entry, the data compared were as follows: admission date; patient demographics (including hospital number); fracture configuration; date and type of surgery; re-operation and date of death. Admission and demographic data were compared to the data held in the hospital administrative systems. Fracture patterns and type of surgery were scrutinised using radiographs stored on the hospital PACS system. Each hospital number on the NHFD dataset was copied into the PACS system in order to view admission, intraoperative (where applicable) and post-operative radiographs. The fracture pattern and operation type were recorded, with the author blinded to the corresponding data on the NHFD dataset. In cases where no intraoperative or post-operative imaging was performed, the patient notes were consulted. Re-operation was confirmed using the same method in combination with the appropriate Office of Population Censuses and Surveys (OPCS) codes. Data comparison was performed independently by two authors and cross-checked for disagreement, which was resolved by joint review.

Electronic health record system

Since October 2014, our centre has used Epic Hyperspace (Epic Systems, Verona, Wisconsin, USA) for all clinical services, with no paper medical records used since. This EHRS includes a built-in hip fracture operation note template which can be used by clinicians to document the procedure. This contains mandatory fields relevant to NHFD data, such as fracture pattern and operation type. Each episode recorded using the EHRS was scrutinised for use of this template.

Statistical analysis

Reliability data were calculated by cross-tabulation and calculation of sensitivity and specificity with confidence intervals (CIs). The sensitivity and specificity of pre-and post-EHRS data were compared simultaneously using the graphical method, as described by Newcombe, adapted for unpaired samples, taking λ = 0.5 (to reflect the equal seriousness of type 1 and type 2 errors in a database) and two-sided confidence level of 95% [10, 11]. This test allows simultaneous comparison of these two parameters with reference to a ‘gold standard’ (in this case, locally held data scrutinised by the authors) allowing an overall conclusion on whether one dataset is superior to another [10]. Comparison of proportions was performed using the ‘N-1’ Chi-squared test, taking a P-value of 0.05 as significant [12, 13]. A Priori calculation of minimum required sample size for each comparison was performed taking α (type 1 error rate) as 0.05 and β (power) of 0.80. To test the association between EHRS operation note template use and data accuracy, a multivariate logistic regression analysis was performed using error as the response variable with month since EHRS introduction and template use as explanatory variables, taking the null hypothesis that template use had no effect on data accuracy. Template use over time was assessed with a logistic regression model taking template use as a response variable and month since EHRS adoption as an explanatory variable. Odds ratios (OR) and 95% CIs were calculated for logistic regression results. Analyses were performed in R version 3.0.2 (R Foundation for Statistical Computing, Vienna, Austria) with the CARET package [14, 15].

Results

Admission and demographics

Fifteen entries from the pre-EHRS dataset were removed from analysis due to lack of available paper notes. Twelve data duplications occurred (eight pre-EHRS, four post-EHRS). Twenty-three entries contained incorrect hospital IDs (9 pre-EHRS, 14 post-EHRS). There were nine incorrect dates of birth, all prior to EHRS introduction.

The remaining data included 914 men and 2,310 women, with mean age 84 years (46–103). Of these, 2,133 were admitted prior to and 1,091 following EHRS introduction. Mean age was the same for both cohorts. Of the pre-EHRS cohort, 70.6% (1506/2133) were women, compared to 73.7% (804/1091) of the post-EHRS cohort (difference = 3.1%, 95% CI −0.2 to 6.3; Chi-squared = 3.4; P = 0.065). Admission dates were all correctly recorded.

Fracture type

Three hundred and forty fractures were coded incorrectly (10.5%), of which 116 (34%) and 224 (66%) were intracapsular and extracapsular, respectively. There was no significant difference in error rate between the pre-and post-EHRS datasets (10.3% vs 10.9%; Chi-squared = 0.3; P = 0.602).

Operation

All dates of surgery were correct. Tables 1 and 2 summarise coding errors in the NHFD from the pre- and post-EHRS periods, respectively. Hybrid total hip arthroplasty (THA) procedures were included in the cemented group for analysis. The coding error rate fell from 23.2% (494/2133) to 7.6% (83/1091) following EHRS introduction (difference = 15.6%, 95% CI 13.4–18.3; Chi-squared = 119.4; P < 0.001). Bipolar uncemented hemiarthroplasty, uncemented THA and ‘endoprosthesis/other’ were excluded from comparative analysis due to the absence of these procedures in corrected datasets. Table 3 compares sensitivity and specificity of operation codes for each dataset. The post-EHRS dataset was superior for the four most common procedures (unipolar cemented hemiarthroplasty, cemented THA, DHS fixation and long intramedullary nail fixation).

Re-operation

All patients recorded in the NHFD as having returned to surgery within 30 days were correctly captured. Eight additional patients from the pre-EHRS dataset returned to theatre within 30 days.

Mortality

In the pre-EHRS dataset, only 46% (45/99) deaths that occurred during the admission for hip fracture were captured by the NHFD. This improved significantly to 91% (112/123) following EHRS introduction (difference = 45.6%, 95% CI 33.9–55.8; Chi-squared 54.9; P < 0.001). Prior to EHRS introduction, of the 109 patients recorded in the NHFD as having died during the initial visit, 64 (59%) had in fact survived. Following EHRS introduction, all 112 recorded deaths in the NHFD were correct (difference = 41%, 95% CI 31.6–50.4; Chi-squared 57.3; P < 0.001; see Supplementary Appendices 1 and 2 in the Supplementary Data are available in Age and Ageing online.) The post-EHRS dataset had superior sensitivity and specificity for mortality (P < 0.001).

Use of operation note template

A total of 61.4% (670/1091) of post-EHRS episodes utilised the hip fracture operation note template. Template usage increased with each successive month following EHRS adoption (OR 1.06; 95% CI 1.05–1.08, Supplementary Figure 1 in the Supplementary Data are available in Age and Ageing online). After controlling for sample month, entries utilising the template were more likely to be error free than those not using the template (OR 2.69; 95% CI 1.92–3.78).

Discussion

Several studies have utilised NHFD data to establish best practice recommendations for the management of hip fractures [16–19]. Thus, data obtained from the NHFD can guide policy change, and as such database integrity is vital. Several surgical databases have been shown to be error-prone, and methods of optimising their validity are valuable in driving improvements in patient care [7, 20, 21]. Previous studies have shown EHRS implementation to improve the quality of information contained within the patient record; however, none to our knowledge have examined the effect of an EHRS on data submitted to a national database [22].

In this study, errors in both operation code and mortality significantly reduced following EHRS implementation. These two areas of coding are important for many reasons. Operation code data within the NHFD represents an unparalleled research resource for direct comparisons between different types of surgery. As the number of fragility hip fracture patients increases, it will be increasingly important to identify subtle differences in outcome between operation types, as even minor differences may have a clinical significance when amplified across such a large patient group [3]. As a large centralised database, the NHFD allows for the detection of such trends, along with comparison between subgroups based on age, fracture type and other factors. The high error rate recorded from our centre prior to introduction of the EHRS (23.2%), if reflective of the database as a whole, suggests valid research using the NHFD would be challenging. After the introduction of the EHRS, errors fell to 7.6%, suggesting that EHRS use could improve the utility of the NFHD as a large-scale audit and research tool. For example, if our centre had investigated outcomes following intramurally nailing using our pre-EHRS dataset, any conclusions drawn would be unreliable due to a 19% coding error rate. By contrast, analysis using our post-EHRS dataset would allow for reliable conclusions due to the low error rate of 2%. This would empower policymakers to use NHFD data as justification for change.

Mortality data are vital not only for identifying avoidable deaths but also in the context of transparency. Since 2016, hospital-specific data have been publicly available from the NHFD in an effort to better inform the public about services in their area [23]. Due to its emotive nature, mortality is likely to be one of the measures most closely scrutinised by third parties, despite its utility in identifying outliers being questionable [24]. Prior to EHRS use, mortality data from our centre lacked sensitivity and specificity. This could conceal trends in urgent need of address and risk reputational damage to the trust. However, our results suggest an EHRS is an effective measure to combat such errors and ensure the validity of publicly available data.

When an operation note template was used, data errors were significantly reduced. Such standardisation of data collection for audit purposes has been shown to increase data accuracy by reducing the number of steps in the data collection process and reducing errors in data transcription from one format (in this case, the patient record), to another (the NHFD website) [9, 25, 26]. However, less than two-thirds of encounters utilised the operation note template, with use steadily increasing over time. This lag time between implementation and uptake of a new healthcare tool is well recognised and has several contributing factors including a lack of perceived benefit, real or imagined technical incompetence and lack of incentive [27, 28]. This study may prove useful to centres adopting an EHRS as it demonstrates benefits that may motivate clinicians to fully engage with new technology.

The main limitation of this study is that it was performed at a single centre. Including other centres would not have allowed for reliable conclusions to be drawn regarding the effect of an EHRS on NHFD data, as ours is the only centre in our region with a fully integrated EHRS. The NHS aims to be paperless at the point of service by 2020; a change which should enable a multicentre approach to further assess the impact of an EHRS on database errors [29].

An additional limitation is the study’s uncontrolled nature. As with many retrospective studies, it is quite possible that other unknown factors contributed differences between the two datasets, such as increasing experience in the health professionals uploading NHFD data.

Another limitation is the difference in time period and hence sample size between the two datasets; the EHRS dataset was approximately half the size of the pre-EHRS dataset. This was primarily due to time constraints; the authors only had access to this centre’s records for a specific period. Encouragingly, our analyses showed no significant demographic differences between the two samples, and the datasets were large enough to achieve adequate statistical power for comparisons.

Our study utilised hospital administration systems, PACS and patient notes as our gold standard for data accuracy. Data extracted from these sources using a standardised method has shown to yield data accuracy approaching 95% [26]. This leaves the possibility of errors in our gold standard dataset, which would subsequently exaggerate the error rate in the NHFD data. In addition, we found no suitable mechanism for identifying any hip fracture admissions which were missed by the NHFD.

We utilised the graphical method for simultaneous comparison of sensitivity and specificity of the datasets [10, 11]. Comparison could have been performed by examining the overall accuracy of the two datasets or by calculating receiver operating characteristic curves and subsequently comparing the area under these curves (AUROC) [30]. However, these methods do not allow for a weighting (λ) to be assigned to the data based on the relative seriousness of type 1 and type 2 errors [10].

In conclusion, introducing an EHRS greatly reduced errors in NHFD data submitted from our centre. This has positive implications for the future of the NHFD as more centres become paperless. However inaccuracies remain, particularly in fracture coding. Use of an operation note template within the EHRS that integrated data collection greatly reduced errors but had poor uptake among clinicians. Mandating the use of such tools at the time of surgery would further improve database integrity. We are in the process of integrating a ‘hard stop’ into the post-operative phase of the hip fracture care pathway which requires the completion of the operation note template prior to patient discharge from recovery.

Acknowledgements: The authors wish to thank Miss Kerri Peacock (BSc), orthogeriatric specialist nurse, Addenbrooke’s Hospital Cambridge, for her help with data extraction for the study.

Declaration of Conflicts of Interest: None.

References