Challenges to electronic clinical quality measurement using third-party platforms in primary care practices: the healthy hearts in the heartland experience

Abstract

BACKGROUND

Since the passage of the 2009 Health Information Technology for Economic and Clinical Health Act and implementation of the electronic health records (EHRs) Incentive Programs, EHR adoption has rapidly climbed to nearly 100% for hospitals and 80% for ambulatory practices.¹ However, widespread adoption has not led to widespread improvements in quality, efficiency, and outcomes,² which may be in part attributable to the challenge with reliably measuring quality of care.³

Numerous studies have described the challenges practices face in calculating and reporting electronic clinical quality measures (eCQMs). These include lack of structured data, low rates of data completeness, and poor data accuracy.^4–8 Small primary care practices face additional challenges in eCQM reporting for myriad reasons, including cost to license additional EHR components to manage eCQMs, limited access to informatics expertise, and limitations of EHR systems.^8,9 These challenges and others hinder the ability of small practices, who tend to have fewer health information technology resources the larger practices and health systems, to accurately calculate and report eCQMs. They also have financial implications under the Merit-based Incentive Payment System (MIPS), the federal quality reporting program passed as part of the Medicare Access and CHIP Reauthorization Act (MACRA).

As part of the Meaningful Use Certification process, EHR vendors have increasingly adopted various standards and implementation specifications designed for information exchange, quality measurement, and reporting.^1,10 For EHR systems with limited eCQM capabilities, standardization potentially enables third-party platforms to consume exported data, calculate eCQMs, and report to the federal government and back to the clinical practice for quality improvement. The Centers for Medicare and Medicaid Services (CMS) certifies several vendors—including private companies and professional societies—for calculating and reporting eCQMs as registries. The growth in third-party platforms reflects an emerging opportunity to support practices in calculating and reporting eCQMs.

The capabilities of third-party platforms to perform eCQM calculations for small practices is contingent on a reliable method of extracting and analyzing EHR data through mechanisms such as standardized EHR data exports, direct database extracts, or open application programing interfaces (APIs). Little is known about the capabilities and limitations of EHRs to enable data access via standard or other mechanisms for third-party platforms in small- to medium-sized, primary care practices.

This work was conducted as part of EvidenceNOW,¹¹ a national cooperative that tests strategies to improve performance of four cardiovascular quality measures: aspirin dosing, blood pressure control, cholesterol management, and smoking cessation. As part of that effort, our group connected practices’ EHRs to an open-source eCQM platform called popHealth,¹² which has been certified by the Office of the National Coordinator for Health IT (ONC). During this process, we identified challenges to using third-party platforms for the calculation of four quality measures that are part of a national effort to improve cardiovascular prevention and care called the Million Hearts^® initiative (Table 1).¹³

METHODS

Sample

Healthy Hearts in the Heartland (H3)¹⁴ is a quality improvement cooperative dedicated to measuring and improving cardiovascular care in small- to medium-sized primary care practices within three states: Illinois, Indiana, and Wisconsin. From November 2015 to October 2016, we enrolled 226 small- to medium-sized practices with an average of 3.4 clinicians per practice. We defined small- to medium-sized practices as those with up to 25 primary care clinicians. Those practices utilized a total of 22 different EHR systems, seven of which we developed connectors to popHealth.

All practices who had one of the seven systems for which we developed connectors were offered to be connected to popHealth. We provided access to a customized version of popHealth for interested practices.

Software—popHealth

Starting with the ONC-certified v4.0.0 release of popHealth (https://github.com/OSEHRA/popHealth/tree/v4.0.0), we extended the system with user interface enhancements specific to our study needs, system enhancements to speed up measure calculations, and API extensions to allow automated data loading and running of calculations.

Data extraction strategies

Given the differences in EHR capabilities, we applied four strategies to export patient-level data from the EHR system in each practice. We applied the strategies in the following order, stopping when we identified a strategy that provided the most complete data set:

1. Direct database extracts—acquire a direct connection to the underlying database system the EHR runs on, and extract the specific fields and tables needed for measure calculations.

2. APIs—a system-level call (typically as a web service) that provided access to patient-level data.

3. Standard export—receive an export of all patient records in a standards-based format, including Consolidated-Clinical Document Architecture (C-CDA) or Continuity of Care Document (CCD),^15,16 which contain administrative, demographic, and clinical data in a universal format. This required the EHR to export all records in a single batch file for retrieval.

4. EHR-based reporting—for EHRs that provide additional reporting capabilities, we generated reports containing patient-level data. This included reports built into the EHR, as well as custom-developed reports if existing reports were not sufficient.

Once we found a successful data export strategy, we then attempted to automate the export or use a manual process to run and download extracts. Using proprietary software (Park Street Solutions; Naperville, IL), we securely uploaded data extracts from practice EHRs to a centralized data server. For each EHR, we established a base extract-transform-load pipeline and customized it as needed for individual practice EHRs. In addition to extracting data from the relevant fields, we also performed the following steps as needed:

1. Converted medication names and descriptions into RxNorm codes;

2. Converted smoking status names and descriptions into the appropriate measure Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT) value set object identifiers;

3. Split blood pressure results into systolic and diastolic components.

Analysis

During the implementation process, we evaluated the various strategies to access the clinical data necessary for each of the four quality measures of interest. The four quality measures required several data types including problem lists, encounter diagnostic codes, medication codes, smoking history, blood pressure measurements, and laboratory results (specifically, cholesterol measurements). For each EHR, we identified the potential options for retrieving patient-level data mentioned above and recorded how each vendor licensed the output mechanisms. We also examined whether we were able to batch individual-level data for output. Batching, which consists of extracting groups of individual-level data at one time in contrast to individually extracting each record, is essential to high-throughput quality measure calculation via third-party platforms. We assessed technical feasibility by review of available EHR documentation, individual evaluation of EHR capabilities, as well as discussions with vendor technical support when possible.

During the implementation process, the technical team (ANK, LVR, PK, and AS) systematically recorded each obstacle encountered during quality measure calculation as appropriate and developed solutions to those when technically feasible. The technical team then presented its categorization schema to the larger study team, which vetted, suggested edits, and ultimately approved a final schema. Two members of the technical team (LVR and AS) reviewed practice-level data and quantified the prevalence of each challenge at the practice-level and by EHR system.

RESULTS

Out of a total of 226 practices, a total of 153 had an EHR system for which we had the technical capabilities to develop popHealth connectors. Of the 153 practices eligible for popHealth connection, a total of 125 practices (81.7%) agreed to be connected to popHealth. Of those who agreed to be connected, we successfully connected a total of 116 practices (92.8%) with seven different EHR systems (Table 2) to popHealth.

The primary reasons for failure to connect were: (1) the lack of a functional, licensed feature in the practice EHR system to generate reports; (2) Inability to batch CCD/C-CDA extracts; (3) Unable to access data from a third-party host; (4) and the practice subsequently declined to allow the connection to be established. We note that we were able to establish a connection to one of the EHR vendors (Vendor G in Table 2) to collect baseline data; however, subsequent attempts to refresh the data failed. This was due to lengthy processing times to export the CCD documents, which made the EHR unusable while running, and could not be scheduled for off-hours. None of the sites that agreed to be connected had implemented any of the remaining 15 EHR systems. Table 3 illustrates the practice characteristics for the practices stratified by connection status. Practice characteristics were overall similar except that a higher percentage of practices that were not connected to PopHealth (unable to be connected or declined) were clinician-owned solo or group practices. Of the 64 clinician-owned solo or group practices, 31 (48.4%) were not connected to popHealth due to either technical limitations (n = 9) or declined (n = 22).

We identified and organized challenges into six categories of barriers to extracting eCQM data from our sample of EHRs: (1) Lack of coded eCQM data; (2) Incorrectly categorized eCQM data; (3) Isosemantic data; (4) Coding that could not be directly evaluated; (5) Errors in date assignment and labeled as historical values; and (6) Inadequate data to assign correct code (Table 4). Challenges were related to the implementation of the EHR vendor system, the lack of specificity of the standards, the measure specifications and associated value sets, and data entry by user. In terms of eCQM calculation, we identified data-related challenges that affected all three aspects of measure calculation: the numerator and denominator; inclusion; and exclusion criteria. “Inadequate data to assign correct codes” was present in two of seven EHR systems, while the other five were present in four to six systems. Within each practice, if a challenge were present, then it usually affected the majority of measures related to eCQM calculation and would likely affect other measures attempting to use the same type of data.

DISCUSSION

In this report, we highlight challenges when using third-party platforms for extracting data from EHRs for eCQM measurement in small- to medium-sized primary care practices. The ability to use third-party platforms is particularly important among these practices because they may utilize EHRs with less functionality and have access to fewer resources for quality measurement and reporting.

The challenges range from connecting with EHR systems themselves to navigating the different data output mechanisms. These output mechanisms each have their own strengths and limitations and can vary by vendor, and at times, within the same vendor but at different practices. We identified common challenges with the EHR data, the EHR systems, the export mechanisms, the measures, and the value sets. Some of these challenges, such as problems with standards (C-CDA and CCD), have been previously described.^15,17 However, to our knowledge, this is one of the first reports on the technical limitations of using third-party platforms for calculating and reporting eCQMs in the era after Meaningful Use Stages 1 and 2, as well as the challenges across the spectrum of quality measurement faced when using a third-party platform.

Although the EHR adoption rate among ambulatory practice has risen to 80%,¹ end-user satisfaction remains mixed for many reasons including poor usability, increased time per patient visit after EHR adoption, lack of clinician trust in EHR data quality and measures, and lack of functionalities to facilitate accurate billing, adherence to clinical guidelines, and care coordination.^4,8,18–21 Third-party platforms have the potential to improve quality of care and end-user satisfaction by facilitating quality reporting, which can be a challenging and resource-intensive task, especially for small practices. However, the effectiveness of these systems will be limited by difficulties accessing and analyzing data required for quality measurement.

As described elsewhere, some challenges may stem from structured data capture mechanisms by the end-user during patient clinic visits,²² but challenges we describe can occur even when the appropriate data are discretely captured at the point of care. These challenges span each stage of quality measurement: data entry at the time of patient visit, data storage in the vendor database, connecting to the vendor system, extracting data in bulk from the system, and running the measure engine on extracted data.

In light of the complexity of these challenges that span across multiple steps in eCQMs, a parsimonious solution is unlikely. Redesigning EHR systems and clinical workflows and developing natural language processing pipelines to better capture and store structured data using standards are essential.^1,23 Developing eCQMs whose data elements reflect the actual data being captured, stored, and extracted in primary care practices may improve the ability to run eCQMs. The standards themselves (such as C-CDA) would benefit from additional modifications to facilitate quality measurement. While all EHRs in our sample were capable of generating C-CDAs or CCDs, we did not use this method for data access in most cases due to incomplete or missing historical data needed for quality measures. We note that the HL7 Fast Healthcare Interoperability Resources standard more adequately supports quality measurement, but its success will require EHR vendors to provide a robust implementation if it will improve upon existing C-CDA and CCD offerings. Furthermore, revising the EHR federal certification process by ONC to improve data capture, storage, and export essential to eCQM may stimulate changes to current EHR vendor systems and standards.

To achieve our goal of using a third-party platform for quality measurement, we had to customize popHealth and develop strategies for data extraction and transformation that varied by EHR system and, at time, at the practice level. This underscores the challenges in developing a single platform that can be used across different EHR systems and different types of practices. Moreover, our analysis highlights how the capabilities to use third-party platforms in clinician-owned solo and group practices may differ from health system-owned clinics or federally qualified health centers. Even with an experienced technical team, we were only able to connect less than half of all eligible clinician-owned practices for either technical reasons or because the practice declined the connection.

The main limitation of this report is that these findings represent the experience of Healthy Hearts in the Heartland Collaborative working with small- to medium-sized practices in the Midwest using a customized, third-party platform and may not be applicable to all EHR systems, practices, or geographic areas. Another limitation is that although we attempted to accurately evaluate the export capabilities for each vendor, we recognize that we may have not have exhaustively identified each potential method for data export given limited access to documentation, systems, and support. However, although other methods to export data may exist within the vendor systems, our approach represents the capabilities of vendor systems in real-world primary care practices identified by a team of experts.

In summary, we have highlighted the challenges of using third-party platforms for quality measurement in small- to medium-sized primary care practices. Clinical workflow and EHR system redesign and wider adoption of improved standards will be essential to improving the ability of primary care practices to receive accurate MACRA-MIPS financial payments tied to quality reporting.

FUNDING

This work was supported by the Agency for Healthcare Research and Quality 5R18HS023921. Dr. Ahmad was in part supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under Award T32HL069771 and the Agency for Healthcare Research and Quality under Award K12HS0323011. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality and the National Heart, Lung, and Blood Institute.

CONTRIBUTORSHIP STATEMENT

This manuscript was initiated and conceptualized by FA, LR, SP, AS, and AK All author contributed to the acquisition, analysis, or interpretation of data. FA and LR drafted the manuscript. All authors critically revised the manuscript and approved the final version.

COMPETING INTERESTS

Andy Schiever was the Chief Technology Officer of Park Street Solutions at the time of conducting this study.

ACKNOWLEDGMENTS

We would to acknowledge all of the members of the Healthy Hearts in the Heartland Collaborative for their contribution to this project.

REFERENCES