https://orcid.org/0000-0003-0744-3923
Abstract
The aim of this systematic review was (i) to assess whether electronic audit and feedback (A&F) is effective in primary care and (ii) to evaluate important features concerning content and delivery of the feedback in primary care, including the use of benchmarks, the frequency of feedback, the cognitive load of feedback and the evidence-based aspects of the feedback.
The MEDLINE, Embase, CINAHL and CENTRAL databases were searched for articles published since 2010 by replicating the search strategy used in the last Cochrane review on A&F.
Two independent reviewers assessed the records for their eligibility, performed the data extraction and evaluated the risk of bias. Our search resulted in 8744 records, including the 140 randomized controlled trials (RCTs) from the last Cochrane Review. The full texts of 431 articles were assessed to determine their eligibility. Finally, 29 articles were included.
Two independent reviewers extracted standard data, data on the effectiveness and outcomes of the interventions, data on the kind of electronic feedback (static versus interactive) and data on the aforementioned feedback features.
Twenty-two studies (76%) showed that electronic A&F was effective. All interventions targeting medication safety, preventive medicine, cholesterol management and depression showed an effect. Approximately 70% of the included studies used benchmarks and high-quality evidence in the content of the feedback. In almost half of the studies, the cognitive load of feedback was not reported. Due to high heterogeneity in the results, no meta-analysis was performed.
This systematic review included 29 articles examining electronic A&F interventions in primary care, and 76% of the interventions were effective. Our findings suggest electronic A&F is effective in primary care for different conditions such as medication safety and preventive medicine. Some of the benefits of electronic A&F include its scalability and the potential to be cost effective. The use of benchmarks as comparators and feedback based on high-quality evidence are widely used and important features of electronic feedback in primary care. However, other important features such as the cognitive load of feedback and the frequency of feedback provision are poorly described in the design of many electronic A&F intervention, indicating that a better description or implementation of these features is needed. Developing a framework or methodology for automated A&F interventions in primary care could be useful for future research.
Introduction
Audit and feedback (A&F) is a well-known healthcare intervention to improve the quality of care that can be defined as ‘any summary, which was delivered to healthcare providers of their clinical performance over a specific period in time’ [1]. A&F has been proven to be effective, but there is no gold standard available for the design and implementation of an A&F intervention [2, 3]. To better understand the development and effectiveness of future A&F interventions certain design elements, hypotheses and suggestions for improvement have been published [4–6]. However, the importance of most of these in the design of an A&F intervention needs to be investigated [5–7]. In addition to the many studies examining why and when A&F is more effective, research is being published on creating tools to facilitate feedback, especially via an electronic medium [8–11]. With the evolution in health information technology, electronic A&F based on data stored in electronic health records (EHRs) offers a promising evolution in A&F interventions [12–15]. Large data repositories and EHR-extractable quality indicators are already available and could be useful for this purpose [16–22].
A previous systematic review investigated the effectiveness of electronic A&F interventions in primary and secondary care [23]. However, the authors only included interventions where electronic A&F was provided with the help of ‘interactive computer interfaces’; such interventions not only give feedback but also allow interaction with the feedback to assist in the clinical decision process [23]. For the purpose of our literature review, this definition is too narrow and excludes less comprehensive and rather static forms of electronically delivered feedback that might be effective in a primary care setting. In addition, little is known about electronic A&F in primary care and more specifically how electronic A&F interventions can be optimized in primary care. Previous work indicated that feedback is a promising tool for quality assessment in primary care but that more research is needed, especially concerning electronic feedback [24]. Moreover, observational research indicated that general physicians (GPs) prefer brief feedback interventions and reports with comparisons and best practice guidelines for quality assessments [25].
The aim of this systematic review was therefore (i) to assess whether electronic A&F is effective for improving health provider performance and healthcare outcomes in primary care and (ii) to evaluate important features concerning content and delivery of electronic feedback in primary care that could optimize the design of future electronic A&F interventions, namely the evidence-based aspect, the cognitive load, the frequency and the use of comparators, as proposed in previous research [25].
Methods
Background
The protocol of this systematic review is described in detail in PROSPERO: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42018089069.
Our inclusion and exclusion criteria and our search strategy were based on the Cochrane review on A&F [3]. Our methods adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [26] (see Appendix 1).
Inclusion and exclusion criteria
Inclusion criteria: participants, interventions, comparators, outcomes and design
The included participants were primary healthcare providers who were responsible for patient care. The interventions studied had to involve electronic A&F alone or as a core element of a multifaceted intervention in primary care. Electronic A&F was defined as ‘any summary, which was delivered electronically, of clinical performance of healthcare over a specified period of time.’ A broad definition was used to investigate both static (without the ability to interact with the feedback) and interactive (with the ability to interact with the feedback) types of electronic feedback. Electronic A&F (alone or as a core element of a multifaceted intervention) was compared with usual care or different types of A&F. The latter comparison is used to evaluate potential differences between the aspects of the content and delivery of the feedback, including the evidence-based aspect, the cognitive load, the frequency and the use of benchmarks. The outcome of interest was the effect of the intervention on improving healthcare provider performance and healthcare outcomes. Finally, the included studies were randomized controlled trials (RCTs; see Appendix 2).
Exclusion criteria
Studies in which real-time feedback was provided during procedural skills were excluded as well as studies that examined feedback on performance with simulated patient interactions or studies in which the term feedback would be best classified as ‘facilitated relay’ of patient-specific clinical information [3]. RCTs that were not conducted in a primary care setting were also excluded. Studies without full-text availability were excluded. Studies were also excluded if they lacked clarity as to whether feedback was delivered electronically (see Appendix 2).
Searches
We searched the following databases: MEDLINE (Ovid) was searched on 25 November 2018, Embase (Elsevier) was searched on 25 November 2018, CINAHL (EBSCO) was searched on 31 October 2018 for trials published since 2010 until the end of October 2018, and the Cochrane Central Register of Controlled Trials (CENTRAL) was searched on 14 February 2019 for studies published since 2010 until the beginning of February 2019 (searched 14 February 2019). The CENTRAL database was searched again at a later date due to technical issues. We searched for trials published since 1 January 2010 to ensure that there was some overlap with the results from the Cochrane review from 2012 and to avoid articles being missed. The search strings are available in Appendix 3.
The 140 RCTs included in the Cochrane review were also added to our search results [3].
Data collection and synthesis
Selection of studies
After removing duplicates, the titles and abstracts were independently screened by two review authors (S.V.D.B. and D.S.) and were classified as inclusion, doubt or exclusion. The full texts of all articles that were classified as doubt and inclusion were obtained. Two review authors, both medical doctors with experience in conducting A&F interventions (S.V.D.B. and D.S.), independently read all full manuscripts and reapplied the inclusion criteria. If there was still no consensus or if doubt remained after reading the full text, a third review author, the director of Cochrane Belgium (P.V.), was consulted to give his opinion.
Data extraction and risk of bias assessment
Two independent reviewers (S.V.D.B. and D.S.) extracted the data and assessed the risk of bias. A data extraction sheet and a list to assess the risk of bias were tailored based on tools provided by Cochrane [27–29]. A separate data extraction file was made for dichotomous and continuous data. Some features of feedback, based on previous observational research and on hypotheses regarding content and delivery, were also incorporated on our data extraction sheet [3–6, 25]. These features were feedback frequency, evidence-based aspect of the feedback (feedback based on evidence-based guidelines: yes, no or unclear), the use of benchmarks as comparisons in the feedback (yes, no or unclear) and the cognitive load of the feedback (feedback with a low cognitive load: yes, no or unclear). Interventions with feedback consisting of many graphs and/or text were considered as having a high cognitive load, while interventions with few graphs and no unnecessary in-depth elements or text were considered as having a low cognitive load.
Disagreements were resolved by discussion. If no consensus was reached, another reviewer (P.V.) was consulted. For each article, standard data were extracted (see Appendices 4 and 5 for the data extraction sheets for continuous and dichotomous outcomes, respectively).
For our risk of bias summary, blinding of participants and personnel (performance bias) was not considered a key domain since the nature of an A&F intervention makes blinding difficult. The risk of performance bias was therefore not used to calculate the summarized risk of bias of the different studies. However, all of the other forms of bias were considered key domains, and if one of them had a high or an unclear risk of bias, the summary was considered as having a high or an uncertain risk of bias, respectively.
Data synthesis
Because no meta-analysis was carried out, the results were described narratively. The hypotheses concerning content and delivery of the feedback were described for each included study, and the type of electronic delivered feedback (static versus interactive) was evaluated. Feedback was defined as static if it was not possible to interact with the electronic feedback and defined as interactive if it was possible to interact with the electronic feedback. The effect of each intervention on improving health provider performance and healthcare outcomes was evaluated. Interventions were considered effective if they had statistically significant results (P < 0.05).
Results
Searches
A total of 12 054 records were identified through database screening. The 140 articles from the Cochrane review were also included, which resulted in a list of 8744 records after removing the duplicates (see Figure 1).
PRISMA flow chart.
Data collection and synthesis
Selection of studies
In total, 8313 records were excluded, mostly because there was no (electronic) A&F intervention or because they were not conducted in a primary care setting. A total of 431 full-text articles were assessed for eligibility, and 402 articles were excluded (see Figure 1). The total number of studies included through database searching was 23 [30–52], and an additional 6 [53–58] articles published before 2010 were included from the Cochrane review published in 2012.
Data extraction and risk of bias assessment
The data extraction indicated that 12 articles (41%) had continuous outcomes, while 17 articles (59%) had dichotomous outcome measures. There was high heterogeneity in the outcome measures of the trials, and a wide range of clinical conditions was targeted by the interventions. Most trials used a cluster RCT design, although 5 studies (17%) used an RCT design. Most interventions included general physicians, but there were 2 trials (7%) including dentists and 1 trial (3.5%) including pharmacists. Patients were often the unit of analysis (19 studies, 65.5%), but some studies used the providers (7 studies, 24%) or the distribution/prescriptions of medication (2 studies, 7%) as the unit of analysis. Finally, one study (3.5%) analyzed both data at the patient and provider levels (see Tables 1 and 2).
There was a high risk of performance bias in 17 of the included studies (59%), while the risk of selection and detection bias was minimal. The risk of both attrition and reporting bias was high in 6 different studies (21%). To summarize, 4 studies (14%) had a low overall risk of bias, 12 studies (41%) had a high risk of bias and 13 studies (45%) had an unclear risk of bias (See Figure 2 and Appendix 6).
Data synthesis
Twenty-two studies (76%) showed a statistically significant effect of the intervention, of which 3 studies (10.5%) only had a partial effect (on only one of multiple co-primary outcomes). Of the 8 studies with interactive electronic feedback, 5 showed an effect of the intervention, while 17 studies out of 21 with static electronic feedback were effective. The data on the different features of electronic feedback showed 17 studies (58.5%) where feedback was provided more than once. In 19 studies (65.5%), the feedback was evidence-based and 20 studies used benchmarks as a comparison in the feedback (69%). Finally, the cognitive load of the feedback was low in 12 studies (41%).
In addition, all electronic A&F interventions targeting medication safety, preventive medicine, cholesterol management and depression were effective. Four studies incorporated all important features of feedback (evidence-based aspect, the cognitive load, the frequency and the use of benchmarks), and three of them showed an effect. Of these Elouafkaoui et al. included all general dentist practices in Scotland and used an A&F intervention based on routinely collected electronic data. In this particular study, subgroup analyses indicated that evidence-based feedback with a low cognitive load showed the greatest impact while the use of benchmarks gave no significant difference [37]. However, Hayashino et al. used the average of the 10% best performers as benchmark according to the Achievable Benchmark of Care method to improve the quality of care for diabetes [41]. Finally, Gerber et al. nearly halved the inappropriate prescribing of broad-spectrum antibiotics to children [45]. (See Tables 1, 2 and 3)
Discussion
Principal findings and comparison with previous work
This systematic review identified 29 articles describing an electronic A&F intervention in primary care. Overall, 22 studies (76%) showed an effect of the intervention on different outcome measures. All interventions targeting medication safety, preventive medicine, cholesterol management and depression were effective. Eight (27.5%) of the electronic feedback interventions were interactive, while 21 studies (72.5%) used static electronic feedback. Approximately 70% of the included studies used benchmarks and high-quality evidence in the content of the feedback. There was high heterogeneity in the results, making a meta-analysis unreliable.
Studies with continuous outcomes
| Study ID (first author, Year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| M. Bahrami, 2003 | CRCT | Compliance with the SIGN guideline | The proportion of patients whose treatment complied with the guideline | Group 1 received a copy of SIGN Guideline and an opportunity to attend a postgraduate education course. Group 2 received A&F. Group 3 received a CAL package. Group 4 received A&F and CAL | Management of unerupted and impacted third molar teeth | Dentist |
| P. Elouafkaoui, 2016 | CRCT | Prescribing antibiotics | Total number of antibiotic items dispensed per 100 NHS treatment claims | Control (no A&F) versus (i) individualized graphical A&F or (ii) individualized graphical A&F plus a written behavior-change message | Infections | Dentist |
| J. S. Gerber, 2013 | CRCT | Prescribing antibiotics | (i) Change in broad-spectrum antibiotic prescribing and (ii) change in antibiotic prescribing for viral infections | One 1-hour on-site clinician education session followed by 1 year of personalized, quarterly A&F of prescribing for bacterial and viral acute respiratory tract infections versus usual practice | Acute respiratory tract infections | GP |
| Y. Hayashino, 2015 | CRCT | Improving the technical quality of diabetes care | The rate of patient drop-out. Despite the process of diabetes care was evaluated | Usual care versus a disease management system of monitoring and providing feedback on the quality of diabetes care, reminders for regular visits and lifestyle modifications | Diabetes care | GP |
| L. G. Hemkens, 2017 | RCT | Prescribing antibiotics | Prescribed DDD of antibiotics to any patient per 100 consultations | Quarterly updated personalized prescription feedback versus physicians in the control group who received no material | Infections | GP |
| T.A. Holt, 2017 | CRCT | Prescribing OACs | Proportion of patients eligible for OACs who were prescribed an OAC at the end of the intervention period | Feedback (reminders) versus usual care | Atrial fibrillation | GP |
| N. H. McAlister, 1986 | CRCT | Hypertension management | Percentage of patients with a diastolic pressure of 90 mm Hg or less at last visit | Usual care versus computer-generated feedback | Hypertension | GP |
| D. R. Murphy, 2015 | CRCT | Reduce delays in diagnostic evaluation for three types of cancer | Time to documented follow-up for colon cancer, prostate cancer and lung cancer | Electronic triggers applied on EHR data repositories versus no electronic triggers | Colon, prostate and lung cancer | Physicians, physicians assistants, nurse practitioners |
| M. S. Patel, 2018 | CRCT | Prescribing statins | The change in the percentage of patients eligible for statin prescription | New statin prescription rates for usual care with primary care providers receiving an active choice intervention with and without peer comparison performance feedback | Hyperlipidemia | Primary care providers |
| Sarafi, A. Nejad, 2016 | RCT | Prescribing behavior of PSs | The monthly total sum of drug usage calculated as the sum of DDD for PSs | Providing performance feedback by traditional postal letters or mobile short text messages compared to usual care | Prescribing parental steroids | GP |
| L. P. Svetkey, 2009 | CRCT | Hypertension management | The change in systolic blood pressure from baseline to 6 months | Physician intervention versus control and/or patient intervention versus control. Physician intervention included internet-based training, self-monitoring, and quarterly feedback reports. Patient intervention included 20 weekly group sessions followed by 12-monthly telephone counseling contacts | Hypertension | Internal medicine or family practice and patients |
| J. Trietsch, 2017 | CRCT | Test ordering and prescribing behavior in primary care | The volumes of tests ordered and drugs prescribed per practice, per 1000 patients, per 6 months | Groups in both trial arms were exposed to the same intervention, but on different clinical topics. Each LQIC in one arm served as an unmatched control for the LQICs in the other arm. Core elements of the intervention are audit and comparative feedback on test ordering and prescribing volumes, dissemination of guidelines and peer review in quality improvement collaboratives moderated by local opinion leaders | Anemia, dyslipidemia, prostate and rheumatic complaints, UTI, chlamydia, diabetes type II, stomach and perimenopausal complaints, thyroid dysfunction | GP |
| Study ID (first author, Year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| M. Bahrami, 2003 | CRCT | Compliance with the SIGN guideline | The proportion of patients whose treatment complied with the guideline | Group 1 received a copy of SIGN Guideline and an opportunity to attend a postgraduate education course. Group 2 received A&F. Group 3 received a CAL package. Group 4 received A&F and CAL | Management of unerupted and impacted third molar teeth | Dentist |
| P. Elouafkaoui, 2016 | CRCT | Prescribing antibiotics | Total number of antibiotic items dispensed per 100 NHS treatment claims | Control (no A&F) versus (i) individualized graphical A&F or (ii) individualized graphical A&F plus a written behavior-change message | Infections | Dentist |
| J. S. Gerber, 2013 | CRCT | Prescribing antibiotics | (i) Change in broad-spectrum antibiotic prescribing and (ii) change in antibiotic prescribing for viral infections | One 1-hour on-site clinician education session followed by 1 year of personalized, quarterly A&F of prescribing for bacterial and viral acute respiratory tract infections versus usual practice | Acute respiratory tract infections | GP |
| Y. Hayashino, 2015 | CRCT | Improving the technical quality of diabetes care | The rate of patient drop-out. Despite the process of diabetes care was evaluated | Usual care versus a disease management system of monitoring and providing feedback on the quality of diabetes care, reminders for regular visits and lifestyle modifications | Diabetes care | GP |
| L. G. Hemkens, 2017 | RCT | Prescribing antibiotics | Prescribed DDD of antibiotics to any patient per 100 consultations | Quarterly updated personalized prescription feedback versus physicians in the control group who received no material | Infections | GP |
| T.A. Holt, 2017 | CRCT | Prescribing OACs | Proportion of patients eligible for OACs who were prescribed an OAC at the end of the intervention period | Feedback (reminders) versus usual care | Atrial fibrillation | GP |
| N. H. McAlister, 1986 | CRCT | Hypertension management | Percentage of patients with a diastolic pressure of 90 mm Hg or less at last visit | Usual care versus computer-generated feedback | Hypertension | GP |
| D. R. Murphy, 2015 | CRCT | Reduce delays in diagnostic evaluation for three types of cancer | Time to documented follow-up for colon cancer, prostate cancer and lung cancer | Electronic triggers applied on EHR data repositories versus no electronic triggers | Colon, prostate and lung cancer | Physicians, physicians assistants, nurse practitioners |
| M. S. Patel, 2018 | CRCT | Prescribing statins | The change in the percentage of patients eligible for statin prescription | New statin prescription rates for usual care with primary care providers receiving an active choice intervention with and without peer comparison performance feedback | Hyperlipidemia | Primary care providers |
| Sarafi, A. Nejad, 2016 | RCT | Prescribing behavior of PSs | The monthly total sum of drug usage calculated as the sum of DDD for PSs | Providing performance feedback by traditional postal letters or mobile short text messages compared to usual care | Prescribing parental steroids | GP |
| L. P. Svetkey, 2009 | CRCT | Hypertension management | The change in systolic blood pressure from baseline to 6 months | Physician intervention versus control and/or patient intervention versus control. Physician intervention included internet-based training, self-monitoring, and quarterly feedback reports. Patient intervention included 20 weekly group sessions followed by 12-monthly telephone counseling contacts | Hypertension | Internal medicine or family practice and patients |
| J. Trietsch, 2017 | CRCT | Test ordering and prescribing behavior in primary care | The volumes of tests ordered and drugs prescribed per practice, per 1000 patients, per 6 months | Groups in both trial arms were exposed to the same intervention, but on different clinical topics. Each LQIC in one arm served as an unmatched control for the LQICs in the other arm. Core elements of the intervention are audit and comparative feedback on test ordering and prescribing volumes, dissemination of guidelines and peer review in quality improvement collaboratives moderated by local opinion leaders | Anemia, dyslipidemia, prostate and rheumatic complaints, UTI, chlamydia, diabetes type II, stomach and perimenopausal complaints, thyroid dysfunction | GP |
CAL, computer-aided learning; CRCT, cluster randomized controlled trial; DDD, defined daily dose; LQIC, local quality improvement collaboratives; NHS, National Health Service; OAC, oral anticoagulant; PS, parenteral steroid; SIGN, Scottish Intercollegiate Guidelines Network; UTI, urinary tract infection.
Studies with continuous outcomes
| Study ID (first author, Year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| M. Bahrami, 2003 | CRCT | Compliance with the SIGN guideline | The proportion of patients whose treatment complied with the guideline | Group 1 received a copy of SIGN Guideline and an opportunity to attend a postgraduate education course. Group 2 received A&F. Group 3 received a CAL package. Group 4 received A&F and CAL | Management of unerupted and impacted third molar teeth | Dentist |
| P. Elouafkaoui, 2016 | CRCT | Prescribing antibiotics | Total number of antibiotic items dispensed per 100 NHS treatment claims | Control (no A&F) versus (i) individualized graphical A&F or (ii) individualized graphical A&F plus a written behavior-change message | Infections | Dentist |
| J. S. Gerber, 2013 | CRCT | Prescribing antibiotics | (i) Change in broad-spectrum antibiotic prescribing and (ii) change in antibiotic prescribing for viral infections | One 1-hour on-site clinician education session followed by 1 year of personalized, quarterly A&F of prescribing for bacterial and viral acute respiratory tract infections versus usual practice | Acute respiratory tract infections | GP |
| Y. Hayashino, 2015 | CRCT | Improving the technical quality of diabetes care | The rate of patient drop-out. Despite the process of diabetes care was evaluated | Usual care versus a disease management system of monitoring and providing feedback on the quality of diabetes care, reminders for regular visits and lifestyle modifications | Diabetes care | GP |
| L. G. Hemkens, 2017 | RCT | Prescribing antibiotics | Prescribed DDD of antibiotics to any patient per 100 consultations | Quarterly updated personalized prescription feedback versus physicians in the control group who received no material | Infections | GP |
| T.A. Holt, 2017 | CRCT | Prescribing OACs | Proportion of patients eligible for OACs who were prescribed an OAC at the end of the intervention period | Feedback (reminders) versus usual care | Atrial fibrillation | GP |
| N. H. McAlister, 1986 | CRCT | Hypertension management | Percentage of patients with a diastolic pressure of 90 mm Hg or less at last visit | Usual care versus computer-generated feedback | Hypertension | GP |
| D. R. Murphy, 2015 | CRCT | Reduce delays in diagnostic evaluation for three types of cancer | Time to documented follow-up for colon cancer, prostate cancer and lung cancer | Electronic triggers applied on EHR data repositories versus no electronic triggers | Colon, prostate and lung cancer | Physicians, physicians assistants, nurse practitioners |
| M. S. Patel, 2018 | CRCT | Prescribing statins | The change in the percentage of patients eligible for statin prescription | New statin prescription rates for usual care with primary care providers receiving an active choice intervention with and without peer comparison performance feedback | Hyperlipidemia | Primary care providers |
| Sarafi, A. Nejad, 2016 | RCT | Prescribing behavior of PSs | The monthly total sum of drug usage calculated as the sum of DDD for PSs | Providing performance feedback by traditional postal letters or mobile short text messages compared to usual care | Prescribing parental steroids | GP |
| L. P. Svetkey, 2009 | CRCT | Hypertension management | The change in systolic blood pressure from baseline to 6 months | Physician intervention versus control and/or patient intervention versus control. Physician intervention included internet-based training, self-monitoring, and quarterly feedback reports. Patient intervention included 20 weekly group sessions followed by 12-monthly telephone counseling contacts | Hypertension | Internal medicine or family practice and patients |
| J. Trietsch, 2017 | CRCT | Test ordering and prescribing behavior in primary care | The volumes of tests ordered and drugs prescribed per practice, per 1000 patients, per 6 months | Groups in both trial arms were exposed to the same intervention, but on different clinical topics. Each LQIC in one arm served as an unmatched control for the LQICs in the other arm. Core elements of the intervention are audit and comparative feedback on test ordering and prescribing volumes, dissemination of guidelines and peer review in quality improvement collaboratives moderated by local opinion leaders | Anemia, dyslipidemia, prostate and rheumatic complaints, UTI, chlamydia, diabetes type II, stomach and perimenopausal complaints, thyroid dysfunction | GP |
| Study ID (first author, Year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| M. Bahrami, 2003 | CRCT | Compliance with the SIGN guideline | The proportion of patients whose treatment complied with the guideline | Group 1 received a copy of SIGN Guideline and an opportunity to attend a postgraduate education course. Group 2 received A&F. Group 3 received a CAL package. Group 4 received A&F and CAL | Management of unerupted and impacted third molar teeth | Dentist |
| P. Elouafkaoui, 2016 | CRCT | Prescribing antibiotics | Total number of antibiotic items dispensed per 100 NHS treatment claims | Control (no A&F) versus (i) individualized graphical A&F or (ii) individualized graphical A&F plus a written behavior-change message | Infections | Dentist |
| J. S. Gerber, 2013 | CRCT | Prescribing antibiotics | (i) Change in broad-spectrum antibiotic prescribing and (ii) change in antibiotic prescribing for viral infections | One 1-hour on-site clinician education session followed by 1 year of personalized, quarterly A&F of prescribing for bacterial and viral acute respiratory tract infections versus usual practice | Acute respiratory tract infections | GP |
| Y. Hayashino, 2015 | CRCT | Improving the technical quality of diabetes care | The rate of patient drop-out. Despite the process of diabetes care was evaluated | Usual care versus a disease management system of monitoring and providing feedback on the quality of diabetes care, reminders for regular visits and lifestyle modifications | Diabetes care | GP |
| L. G. Hemkens, 2017 | RCT | Prescribing antibiotics | Prescribed DDD of antibiotics to any patient per 100 consultations | Quarterly updated personalized prescription feedback versus physicians in the control group who received no material | Infections | GP |
| T.A. Holt, 2017 | CRCT | Prescribing OACs | Proportion of patients eligible for OACs who were prescribed an OAC at the end of the intervention period | Feedback (reminders) versus usual care | Atrial fibrillation | GP |
| N. H. McAlister, 1986 | CRCT | Hypertension management | Percentage of patients with a diastolic pressure of 90 mm Hg or less at last visit | Usual care versus computer-generated feedback | Hypertension | GP |
| D. R. Murphy, 2015 | CRCT | Reduce delays in diagnostic evaluation for three types of cancer | Time to documented follow-up for colon cancer, prostate cancer and lung cancer | Electronic triggers applied on EHR data repositories versus no electronic triggers | Colon, prostate and lung cancer | Physicians, physicians assistants, nurse practitioners |
| M. S. Patel, 2018 | CRCT | Prescribing statins | The change in the percentage of patients eligible for statin prescription | New statin prescription rates for usual care with primary care providers receiving an active choice intervention with and without peer comparison performance feedback | Hyperlipidemia | Primary care providers |
| Sarafi, A. Nejad, 2016 | RCT | Prescribing behavior of PSs | The monthly total sum of drug usage calculated as the sum of DDD for PSs | Providing performance feedback by traditional postal letters or mobile short text messages compared to usual care | Prescribing parental steroids | GP |
| L. P. Svetkey, 2009 | CRCT | Hypertension management | The change in systolic blood pressure from baseline to 6 months | Physician intervention versus control and/or patient intervention versus control. Physician intervention included internet-based training, self-monitoring, and quarterly feedback reports. Patient intervention included 20 weekly group sessions followed by 12-monthly telephone counseling contacts | Hypertension | Internal medicine or family practice and patients |
| J. Trietsch, 2017 | CRCT | Test ordering and prescribing behavior in primary care | The volumes of tests ordered and drugs prescribed per practice, per 1000 patients, per 6 months | Groups in both trial arms were exposed to the same intervention, but on different clinical topics. Each LQIC in one arm served as an unmatched control for the LQICs in the other arm. Core elements of the intervention are audit and comparative feedback on test ordering and prescribing volumes, dissemination of guidelines and peer review in quality improvement collaboratives moderated by local opinion leaders | Anemia, dyslipidemia, prostate and rheumatic complaints, UTI, chlamydia, diabetes type II, stomach and perimenopausal complaints, thyroid dysfunction | GP |
CAL, computer-aided learning; CRCT, cluster randomized controlled trial; DDD, defined daily dose; LQIC, local quality improvement collaboratives; NHS, National Health Service; OAC, oral anticoagulant; PS, parenteral steroid; SIGN, Scottish Intercollegiate Guidelines Network; UTI, urinary tract infection.
Studies with dichotomous outcomes
| Study ID (first author, year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| O. P. Almeida, 2012 | CRCT | The care for patients with depression and self-harm behavior | A composite measure of clinically significant depression (Patient Health Questionnaire score ≥ 10) or self-harm behavior (suicide thoughts or attempt during the previous 12 months) | The intervention consisted of a practice audit with personalized automated audit feedback, printed educational material, and 6-monthly educational newsletters.Control physicians completed a practice audit without individualized feedback and 6-monthly newsletters describing the progress of the study | Depression | GP |
| A. J. Avery, 2012 | CRCT | Medication errors | The proportions of patients at 6 months after the intervention who had had any of three clinically important errors | Computer-generated simple feedback (control) versus a pharmacist-led information technology intervention, composed of feedback, educational outreach, and dedicated support | Medication safety | GP |
| B. Bonevski, 1999 | RCT | Preventive medicine | Smoking and benzodiazepine use sensitivity, specificity, and overall accuracy and whether blood pressure and cholesterol screening levels were obtained | Intervention received a computerized feedback system; control group was given usual care | Preventive medicine | GP |
| C. A. Estrada, 2011 | CRCT | Improving diabetes control | ‘Acceptable control’ and ‘optimal control’ of diabetes | A multi-component intervention including Web-based CME, performance feedback and quality improvement tools versus usual care | Diabetes | Primary care physicians |
| T. L. Guldberg, 2011 | CRCT | Quality of type 2 diabetes care | Prescriptions for type 2 diabetes, measuring of HbA1c and cholesterol and visits to ophthalmologists | To receive or not to receive electronic feedback on quality of care | Type 2 diabetes | GP |
| B. Guthrie, 2016 | CRCT | Safety of prescribing | Proportion of patients included in one or more of the defined six individual secondary outcomes (denominator) who receive any high risk prescription (numerator) | Three arms: ‘usual care’ (= emailed educational material with support); usual care plus feedback on practice’s high risk prescribing; usual care plus the same feedback incorporating a behavioral change component | Safety of prescribing | GP |
| W. Y. Lim, 2018 | CRCT | Prescribing medication | The percentage of prescriptions with at least one error (error versus no error) |
| Errors in prescribing | Primary care prescribers |
| J. A. Linder, 2010 | CRCT | Prescribing antibiotic | The primary outcome was the intent-to-intervene antibiotic prescribing rate for acute respiratory infection visits | The ARI Quality Dashboard, an EHR-based feedback system versus usual care | Acute respiratory infections | Primary care physicians |
| J. W. Mold, 2008 | RCT | Preventive medicine | The number of practices implementing one or more evidence-based processes and the total number of processes implemented | Intervention practices received performance feedback, academic detailing, a practice facilitator, and computer support to feedback and benchmarking (= control) | Preventive medicine | Clinicians |
| G. Ogedegbe, 2014 | CRCT | Blood Pressure control | The rate of BP control at 12 months, defined as mean BP < 140/90 mm Hg (or mean BP < 130/80 mm Hg for those with diabetes mellitus or kidney disease) | Intervention patients received education, home BP monitoring and lifestyle counseling. Intervention physicians attended hypertension case rounds and received feedback on their patients’ home BP readings and chart audits. Patients and physicians at the usual care sites received patient education material and hypertension treatment guidelines, respectively | Hypertension | GP |
| S. Ornstein, 2010 | CRCT | CRC screening | Proportion of active patients up to date with CRC screening and having screening recommended within past year among those not up to date | A quality improvement intervention combining EHR based A&F, academic detailing and participatory planning, and ‘best-practice’ dissemination on CRC screening versus usual care | Colorectal cancer | Primary care physicians |
| G. A. Pape, 2011 | CRCT | Cholesterol management in diabetes mellitus | Proportion of participants in each arm achieving a target LDL level of 100 mg/dL or lower | The intervention included remote physician–pharmacist team-based care focused on cholesterol management in DM versus control. All clinicians in the study had access to automated DM-related point-of-care prompts, a Web-based registry, and performance feedback with benchmarking | Cholesterol management in diabetes mellitus | Family practice and internal medicine physicians |
| D. Peiris, 2015 | CRCT | Cardiovascular disease risk management |
| The intervention arm consisted of a computer-guided QI intervention comprising point-of care electronic decision support, A&F tools, and clinical workforce training versus usual care | Cardiovascular disease risk management | GP |
| I. Urbiztondo, 2017 | CRCT | Prescribing antibiotics | The change in the proportion of patients treated with antibiotics for respiratory tract infection | Intervention (evidence-based online feedback) versus control (no exposure to the evidence-based online feedback) | respiratory tract infections | GP |
| D. Vinereanu, 2017 | CRCT | Use of oral anticoagulant medication in atrial fibrillation | The change in the proportion of patients treated with oral anticoagulants | Intervention consisting of two components (education and regular monitoring and feedback) versus usual care | Atrial fibrillation | Health care providers |
| W. C. Wadland, 2007 | CRCT | Smoking cessation | Changes in clinician referrals in both intervention and control groups | Comparing the impact of 6 quarterly feedback reports (intervention) with that of general reminders (control) | Smoking cessation | Clinicians |
| N. Winslade, 2016 | RCT | Provision of professional services and the quality of patients’ medication use | The number of hypertension/asthma services billed per pharmacy and percentage of dispensing to nonadherent patients over the 12 months postintervention | Pharmacy-specific performance feedback reports versus no feedback reports | Asthma and hypertension | Pharmacist |
| Study ID (first author, year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| O. P. Almeida, 2012 | CRCT | The care for patients with depression and self-harm behavior | A composite measure of clinically significant depression (Patient Health Questionnaire score ≥ 10) or self-harm behavior (suicide thoughts or attempt during the previous 12 months) | The intervention consisted of a practice audit with personalized automated audit feedback, printed educational material, and 6-monthly educational newsletters.Control physicians completed a practice audit without individualized feedback and 6-monthly newsletters describing the progress of the study | Depression | GP |
| A. J. Avery, 2012 | CRCT | Medication errors | The proportions of patients at 6 months after the intervention who had had any of three clinically important errors | Computer-generated simple feedback (control) versus a pharmacist-led information technology intervention, composed of feedback, educational outreach, and dedicated support | Medication safety | GP |
| B. Bonevski, 1999 | RCT | Preventive medicine | Smoking and benzodiazepine use sensitivity, specificity, and overall accuracy and whether blood pressure and cholesterol screening levels were obtained | Intervention received a computerized feedback system; control group was given usual care | Preventive medicine | GP |
| C. A. Estrada, 2011 | CRCT | Improving diabetes control | ‘Acceptable control’ and ‘optimal control’ of diabetes | A multi-component intervention including Web-based CME, performance feedback and quality improvement tools versus usual care | Diabetes | Primary care physicians |
| T. L. Guldberg, 2011 | CRCT | Quality of type 2 diabetes care | Prescriptions for type 2 diabetes, measuring of HbA1c and cholesterol and visits to ophthalmologists | To receive or not to receive electronic feedback on quality of care | Type 2 diabetes | GP |
| B. Guthrie, 2016 | CRCT | Safety of prescribing | Proportion of patients included in one or more of the defined six individual secondary outcomes (denominator) who receive any high risk prescription (numerator) | Three arms: ‘usual care’ (= emailed educational material with support); usual care plus feedback on practice’s high risk prescribing; usual care plus the same feedback incorporating a behavioral change component | Safety of prescribing | GP |
| W. Y. Lim, 2018 | CRCT | Prescribing medication | The percentage of prescriptions with at least one error (error versus no error) |
| Errors in prescribing | Primary care prescribers |
| J. A. Linder, 2010 | CRCT | Prescribing antibiotic | The primary outcome was the intent-to-intervene antibiotic prescribing rate for acute respiratory infection visits | The ARI Quality Dashboard, an EHR-based feedback system versus usual care | Acute respiratory infections | Primary care physicians |
| J. W. Mold, 2008 | RCT | Preventive medicine | The number of practices implementing one or more evidence-based processes and the total number of processes implemented | Intervention practices received performance feedback, academic detailing, a practice facilitator, and computer support to feedback and benchmarking (= control) | Preventive medicine | Clinicians |
| G. Ogedegbe, 2014 | CRCT | Blood Pressure control | The rate of BP control at 12 months, defined as mean BP < 140/90 mm Hg (or mean BP < 130/80 mm Hg for those with diabetes mellitus or kidney disease) | Intervention patients received education, home BP monitoring and lifestyle counseling. Intervention physicians attended hypertension case rounds and received feedback on their patients’ home BP readings and chart audits. Patients and physicians at the usual care sites received patient education material and hypertension treatment guidelines, respectively | Hypertension | GP |
| S. Ornstein, 2010 | CRCT | CRC screening | Proportion of active patients up to date with CRC screening and having screening recommended within past year among those not up to date | A quality improvement intervention combining EHR based A&F, academic detailing and participatory planning, and ‘best-practice’ dissemination on CRC screening versus usual care | Colorectal cancer | Primary care physicians |
| G. A. Pape, 2011 | CRCT | Cholesterol management in diabetes mellitus | Proportion of participants in each arm achieving a target LDL level of 100 mg/dL or lower | The intervention included remote physician–pharmacist team-based care focused on cholesterol management in DM versus control. All clinicians in the study had access to automated DM-related point-of-care prompts, a Web-based registry, and performance feedback with benchmarking | Cholesterol management in diabetes mellitus | Family practice and internal medicine physicians |
| D. Peiris, 2015 | CRCT | Cardiovascular disease risk management |
| The intervention arm consisted of a computer-guided QI intervention comprising point-of care electronic decision support, A&F tools, and clinical workforce training versus usual care | Cardiovascular disease risk management | GP |
| I. Urbiztondo, 2017 | CRCT | Prescribing antibiotics | The change in the proportion of patients treated with antibiotics for respiratory tract infection | Intervention (evidence-based online feedback) versus control (no exposure to the evidence-based online feedback) | respiratory tract infections | GP |
| D. Vinereanu, 2017 | CRCT | Use of oral anticoagulant medication in atrial fibrillation | The change in the proportion of patients treated with oral anticoagulants | Intervention consisting of two components (education and regular monitoring and feedback) versus usual care | Atrial fibrillation | Health care providers |
| W. C. Wadland, 2007 | CRCT | Smoking cessation | Changes in clinician referrals in both intervention and control groups | Comparing the impact of 6 quarterly feedback reports (intervention) with that of general reminders (control) | Smoking cessation | Clinicians |
| N. Winslade, 2016 | RCT | Provision of professional services and the quality of patients’ medication use | The number of hypertension/asthma services billed per pharmacy and percentage of dispensing to nonadherent patients over the 12 months postintervention | Pharmacy-specific performance feedback reports versus no feedback reports | Asthma and hypertension | Pharmacist |
ACE-I, angiotensin-converting enzyme inhibitor; ARI, acute respiratory infection; BP, blood pressure; CME, continuing medical education; CRC, colorectal cancer; CRCT, cluster randomized controlled trial; CVD, cardiovascular disease; DM, diabetes mellitus; LDL, low-density lipoprotein; NSAID, nonsteroidal anti-inflammatory drug.
Studies with dichotomous outcomes
| Study ID (first author, year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| O. P. Almeida, 2012 | CRCT | The care for patients with depression and self-harm behavior | A composite measure of clinically significant depression (Patient Health Questionnaire score ≥ 10) or self-harm behavior (suicide thoughts or attempt during the previous 12 months) | The intervention consisted of a practice audit with personalized automated audit feedback, printed educational material, and 6-monthly educational newsletters.Control physicians completed a practice audit without individualized feedback and 6-monthly newsletters describing the progress of the study | Depression | GP |
| A. J. Avery, 2012 | CRCT | Medication errors | The proportions of patients at 6 months after the intervention who had had any of three clinically important errors | Computer-generated simple feedback (control) versus a pharmacist-led information technology intervention, composed of feedback, educational outreach, and dedicated support | Medication safety | GP |
| B. Bonevski, 1999 | RCT | Preventive medicine | Smoking and benzodiazepine use sensitivity, specificity, and overall accuracy and whether blood pressure and cholesterol screening levels were obtained | Intervention received a computerized feedback system; control group was given usual care | Preventive medicine | GP |
| C. A. Estrada, 2011 | CRCT | Improving diabetes control | ‘Acceptable control’ and ‘optimal control’ of diabetes | A multi-component intervention including Web-based CME, performance feedback and quality improvement tools versus usual care | Diabetes | Primary care physicians |
| T. L. Guldberg, 2011 | CRCT | Quality of type 2 diabetes care | Prescriptions for type 2 diabetes, measuring of HbA1c and cholesterol and visits to ophthalmologists | To receive or not to receive electronic feedback on quality of care | Type 2 diabetes | GP |
| B. Guthrie, 2016 | CRCT | Safety of prescribing | Proportion of patients included in one or more of the defined six individual secondary outcomes (denominator) who receive any high risk prescription (numerator) | Three arms: ‘usual care’ (= emailed educational material with support); usual care plus feedback on practice’s high risk prescribing; usual care plus the same feedback incorporating a behavioral change component | Safety of prescribing | GP |
| W. Y. Lim, 2018 | CRCT | Prescribing medication | The percentage of prescriptions with at least one error (error versus no error) |
| Errors in prescribing | Primary care prescribers |
| J. A. Linder, 2010 | CRCT | Prescribing antibiotic | The primary outcome was the intent-to-intervene antibiotic prescribing rate for acute respiratory infection visits | The ARI Quality Dashboard, an EHR-based feedback system versus usual care | Acute respiratory infections | Primary care physicians |
| J. W. Mold, 2008 | RCT | Preventive medicine | The number of practices implementing one or more evidence-based processes and the total number of processes implemented | Intervention practices received performance feedback, academic detailing, a practice facilitator, and computer support to feedback and benchmarking (= control) | Preventive medicine | Clinicians |
| G. Ogedegbe, 2014 | CRCT | Blood Pressure control | The rate of BP control at 12 months, defined as mean BP < 140/90 mm Hg (or mean BP < 130/80 mm Hg for those with diabetes mellitus or kidney disease) | Intervention patients received education, home BP monitoring and lifestyle counseling. Intervention physicians attended hypertension case rounds and received feedback on their patients’ home BP readings and chart audits. Patients and physicians at the usual care sites received patient education material and hypertension treatment guidelines, respectively | Hypertension | GP |
| S. Ornstein, 2010 | CRCT | CRC screening | Proportion of active patients up to date with CRC screening and having screening recommended within past year among those not up to date | A quality improvement intervention combining EHR based A&F, academic detailing and participatory planning, and ‘best-practice’ dissemination on CRC screening versus usual care | Colorectal cancer | Primary care physicians |
| G. A. Pape, 2011 | CRCT | Cholesterol management in diabetes mellitus | Proportion of participants in each arm achieving a target LDL level of 100 mg/dL or lower | The intervention included remote physician–pharmacist team-based care focused on cholesterol management in DM versus control. All clinicians in the study had access to automated DM-related point-of-care prompts, a Web-based registry, and performance feedback with benchmarking | Cholesterol management in diabetes mellitus | Family practice and internal medicine physicians |
| D. Peiris, 2015 | CRCT | Cardiovascular disease risk management |
| The intervention arm consisted of a computer-guided QI intervention comprising point-of care electronic decision support, A&F tools, and clinical workforce training versus usual care | Cardiovascular disease risk management | GP |
| I. Urbiztondo, 2017 | CRCT | Prescribing antibiotics | The change in the proportion of patients treated with antibiotics for respiratory tract infection | Intervention (evidence-based online feedback) versus control (no exposure to the evidence-based online feedback) | respiratory tract infections | GP |
| D. Vinereanu, 2017 | CRCT | Use of oral anticoagulant medication in atrial fibrillation | The change in the proportion of patients treated with oral anticoagulants | Intervention consisting of two components (education and regular monitoring and feedback) versus usual care | Atrial fibrillation | Health care providers |
| W. C. Wadland, 2007 | CRCT | Smoking cessation | Changes in clinician referrals in both intervention and control groups | Comparing the impact of 6 quarterly feedback reports (intervention) with that of general reminders (control) | Smoking cessation | Clinicians |
| N. Winslade, 2016 | RCT | Provision of professional services and the quality of patients’ medication use | The number of hypertension/asthma services billed per pharmacy and percentage of dispensing to nonadherent patients over the 12 months postintervention | Pharmacy-specific performance feedback reports versus no feedback reports | Asthma and hypertension | Pharmacist |
| Study ID (first author, year) | Study design | Type of targeted behavior | Primary outcomes | Type of interventions compared | Clinical condition | Targeted health professional |
|---|---|---|---|---|---|---|
| O. P. Almeida, 2012 | CRCT | The care for patients with depression and self-harm behavior | A composite measure of clinically significant depression (Patient Health Questionnaire score ≥ 10) or self-harm behavior (suicide thoughts or attempt during the previous 12 months) | The intervention consisted of a practice audit with personalized automated audit feedback, printed educational material, and 6-monthly educational newsletters.Control physicians completed a practice audit without individualized feedback and 6-monthly newsletters describing the progress of the study | Depression | GP |
| A. J. Avery, 2012 | CRCT | Medication errors | The proportions of patients at 6 months after the intervention who had had any of three clinically important errors | Computer-generated simple feedback (control) versus a pharmacist-led information technology intervention, composed of feedback, educational outreach, and dedicated support | Medication safety | GP |
| B. Bonevski, 1999 | RCT | Preventive medicine | Smoking and benzodiazepine use sensitivity, specificity, and overall accuracy and whether blood pressure and cholesterol screening levels were obtained | Intervention received a computerized feedback system; control group was given usual care | Preventive medicine | GP |
| C. A. Estrada, 2011 | CRCT | Improving diabetes control | ‘Acceptable control’ and ‘optimal control’ of diabetes | A multi-component intervention including Web-based CME, performance feedback and quality improvement tools versus usual care | Diabetes | Primary care physicians |
| T. L. Guldberg, 2011 | CRCT | Quality of type 2 diabetes care | Prescriptions for type 2 diabetes, measuring of HbA1c and cholesterol and visits to ophthalmologists | To receive or not to receive electronic feedback on quality of care | Type 2 diabetes | GP |
| B. Guthrie, 2016 | CRCT | Safety of prescribing | Proportion of patients included in one or more of the defined six individual secondary outcomes (denominator) who receive any high risk prescription (numerator) | Three arms: ‘usual care’ (= emailed educational material with support); usual care plus feedback on practice’s high risk prescribing; usual care plus the same feedback incorporating a behavioral change component | Safety of prescribing | GP |
| W. Y. Lim, 2018 | CRCT | Prescribing medication | The percentage of prescriptions with at least one error (error versus no error) |
| Errors in prescribing | Primary care prescribers |
| J. A. Linder, 2010 | CRCT | Prescribing antibiotic | The primary outcome was the intent-to-intervene antibiotic prescribing rate for acute respiratory infection visits | The ARI Quality Dashboard, an EHR-based feedback system versus usual care | Acute respiratory infections | Primary care physicians |
| J. W. Mold, 2008 | RCT | Preventive medicine | The number of practices implementing one or more evidence-based processes and the total number of processes implemented | Intervention practices received performance feedback, academic detailing, a practice facilitator, and computer support to feedback and benchmarking (= control) | Preventive medicine | Clinicians |
| G. Ogedegbe, 2014 | CRCT | Blood Pressure control | The rate of BP control at 12 months, defined as mean BP < 140/90 mm Hg (or mean BP < 130/80 mm Hg for those with diabetes mellitus or kidney disease) | Intervention patients received education, home BP monitoring and lifestyle counseling. Intervention physicians attended hypertension case rounds and received feedback on their patients’ home BP readings and chart audits. Patients and physicians at the usual care sites received patient education material and hypertension treatment guidelines, respectively | Hypertension | GP |
| S. Ornstein, 2010 | CRCT | CRC screening | Proportion of active patients up to date with CRC screening and having screening recommended within past year among those not up to date | A quality improvement intervention combining EHR based A&F, academic detailing and participatory planning, and ‘best-practice’ dissemination on CRC screening versus usual care | Colorectal cancer | Primary care physicians |
| G. A. Pape, 2011 | CRCT | Cholesterol management in diabetes mellitus | Proportion of participants in each arm achieving a target LDL level of 100 mg/dL or lower | The intervention included remote physician–pharmacist team-based care focused on cholesterol management in DM versus control. All clinicians in the study had access to automated DM-related point-of-care prompts, a Web-based registry, and performance feedback with benchmarking | Cholesterol management in diabetes mellitus | Family practice and internal medicine physicians |
| D. Peiris, 2015 | CRCT | Cardiovascular disease risk management |
| The intervention arm consisted of a computer-guided QI intervention comprising point-of care electronic decision support, A&F tools, and clinical workforce training versus usual care | Cardiovascular disease risk management | GP |
| I. Urbiztondo, 2017 | CRCT | Prescribing antibiotics | The change in the proportion of patients treated with antibiotics for respiratory tract infection | Intervention (evidence-based online feedback) versus control (no exposure to the evidence-based online feedback) | respiratory tract infections | GP |
| D. Vinereanu, 2017 | CRCT | Use of oral anticoagulant medication in atrial fibrillation | The change in the proportion of patients treated with oral anticoagulants | Intervention consisting of two components (education and regular monitoring and feedback) versus usual care | Atrial fibrillation | Health care providers |
| W. C. Wadland, 2007 | CRCT | Smoking cessation | Changes in clinician referrals in both intervention and control groups | Comparing the impact of 6 quarterly feedback reports (intervention) with that of general reminders (control) | Smoking cessation | Clinicians |
| N. Winslade, 2016 | RCT | Provision of professional services and the quality of patients’ medication use | The number of hypertension/asthma services billed per pharmacy and percentage of dispensing to nonadherent patients over the 12 months postintervention | Pharmacy-specific performance feedback reports versus no feedback reports | Asthma and hypertension | Pharmacist |
ACE-I, angiotensin-converting enzyme inhibitor; ARI, acute respiratory infection; BP, blood pressure; CME, continuing medical education; CRC, colorectal cancer; CRCT, cluster randomized controlled trial; CVD, cardiovascular disease; DM, diabetes mellitus; LDL, low-density lipoprotein; NSAID, nonsteroidal anti-inflammatory drug.
Our findings suggest that electronic A&F is effective for improving the quality of care for different conditions in primary care such as medication safety and preventive medicine. For these conditions, the A&F interventions mostly targeted processes of care instead of outcomes. This could be a potential explanation for their effectiveness because process measures are directly actionable while outcome measures are affected by different factors that are beyond the control of health providers [22, 48, 59]. Thus, electronic A&F interventions targeting processes of care may be more effective, as the health professionals receiving the feedback also have the ability to implement improvements. Furthermore, our review indicated some of the benefits of electronic A&F in primary care: (i) A&F can be deployed (inter)nationally, drastically increasing the number of patients for whom quality of care can be improved [32, 33, 37], (ii) A&F can be cost effective [30, 52] and (iii) A&F can target many different conditions and procedures [32, 46, 47, 50].
Risk of bias summary.
However, our review also confirms that there is insufficient research on implementation to further the field and build on existing knowledge [60]. Previous work showed that feedback is best when it is provided more than once, and our findings indicate that this was only the case in 58.5% of the included studies. However, the use of benchmarks and evidence-based feedback in approximately 70% of the electronic feedback interventions shows a promising trend in the implementation of these 2 hypotheses on the content and delivery of feedback. After all, observational research has shown that GPs prefer brief feedback interventions and reports with comparisons and best practice guidelines [25]. In addition, according to the recent Clinical Performance Feedback Intervention Theory, the use of benchmarks can improve feedback by comparing and motivating feedback recipients [7]. Finally, only 41% of the included studies used feedback with a low cognitive load, making this hypothesis the least common to be implemented among the 4 hypotheses investigated herein regarding the content and delivery of feedback.
Although electronic A&F was studied extensively in primary care, a meta-analysis to pool the results and produce generalizable data was not feasible. This emphasizes the difficulties in designing complex healthcare interventions and indicates the need for a framework and a well-defined research agenda when developing electronic A&F trials so that interventions can be reproduced and compared [60–62]. Designing a methodology for developing generalizable automated A&F interventions in primary care could be useful since automated quality assessment based on EHRs offers promising prospects if the challenges are answered [15].
Large data repositories, such as those of the Dutch NIVEL, the British Royal College of General Practitioners (RGCP) Research and Surveillance Centre (RSC) network, and the Belgian INTEGO database, have already been available for many years in primary care [17, 63, 64]. Using the facilities of these institutes in a well-designed trial with a standardized methodology could address some of the problems in evaluating the effectiveness and features of electronic A&F interventions. In this respect, recent research indicates the need for an evolution from a two-arm trial of A&F versus control to head-to-head trials of various A&F variants to measure small differences in the effectiveness of different A&F features [65]. Such trials need to be sufficiently powered, requiring large sample sizes that could be provided by these large primary care data repositories [65]. However, further research is necessary to develop a methodology for automated and EHR-based A&F interventions in primary care. Designing and using a standardized methodology to create automated A&F interventions based on EHR data could enable comparisons of future electronic A&F interventions and enable investigations of different features of interventions, thus advancing the field of A&F research.
Feedback features
| First author | Year | Unit of analysis | Frequency of the feedback | Evidence-based feedback | Benchmarks | Low cognitive load | Electronic feedback | Effect of intervention | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | O. P. Almeida | 2012 | Patients | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 2 | A. J. Avery | 2012 | Patients | Less than monthly | Unclear | Unclear | Unclear | Static | Yes |
| 3 | B. Bonevski | 1999 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 4 | C. A. Estrada | 2011 | Patients | Less than monthly | Unclear | Yes | Unclear | Interactive | No |
| 5 | T. L. Guldberg | 2011 | Patients | Unclear | Yes | Yes | Yes | Interactive | Yes |
| 6 | B. Guthrie | 2016 | Patients | Less than monthly | Yes | Yes | No | Static | Yes |
| 7 | W. Y. Lim | 2018 | Number of prescriptions | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 8 | J. A. Linder | 2010 | Providers | Monthly | Yes | Yes | Yes | Interactive | No |
| 9 | J. W. Mold | 2008 | Providers | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 10 | G. Ogedegbe | 2014 | Patient | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 11 | S. Ornstein | 2010 | Patients | Unclear | No | Unclear | Unclear | Static | Yes |
| 12 | G. A. Pape | 2011 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 13 | D. Peiris | 2015 | Patient-level data analysis | Unclear | Yes | Yes | Yes | Interactive | Yes (partially) |
| 14 | I. Urbiztondo | 2017 | Individual data at patient and GP level | Weekly | Yes | Unclear | Unclear | Static | Yes |
| 15 | D. Vinereanu | 2017 | Patients | Unclear | Unclear | Unclear | Unclear | Static | Yes |
| 16 | W. C. Wadland | 2007 | Providers | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 17 | N. Winslade | 2016 | Number of dispensings | Once | Yes | Yes | No | Static | Yes (partially) |
| 18 | M. Bahrami | 2003 | Patients | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 19 | P. Elouafkaoui, | 2016 | Providers | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 20 | J. S. Gerber | 2013 | Patients | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 21 | Y. Hayashino | 2015 | Patients | Monthly | Yes | Yes | Yes | Static | Yes |
| 22 | L. G. Hemkens | 2017 | Providers | Less than monthly | Yes | Yes | No | Static | No |
| 23 | T.A. Holt | 2017 | Patients | Unclear | Yes | No | Yes | Interactive | No |
| 24 | N. H. McAlister | 1986 | Patients | Unclear | Yes | Yes | Yes | Static | Yes |
| 25 | D. R. Murphy | 2015 | Patients | Monthly | Unclear | No | Unclear | Static | Yes |
| 26 | M. S. Patel | 2018 | Patients | Monthly | Yes | Yes | Unclear | Interactive | Yes |
| 27 | S. A. Nejad | 2016 | Providers | Less than monthly | Unclear | Unclear | Yes | Static | Yes |
| 28 | L. P. Svetkey | 2009 | Patients | Less than monthly | Yes | Yes | Unclear | Static | Yes (partially) |
| 29 | J. Trietsch | 2017 | Providers | Unclear | Yes | Yes | Yes | Static | No |
| First author | Year | Unit of analysis | Frequency of the feedback | Evidence-based feedback | Benchmarks | Low cognitive load | Electronic feedback | Effect of intervention | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | O. P. Almeida | 2012 | Patients | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 2 | A. J. Avery | 2012 | Patients | Less than monthly | Unclear | Unclear | Unclear | Static | Yes |
| 3 | B. Bonevski | 1999 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 4 | C. A. Estrada | 2011 | Patients | Less than monthly | Unclear | Yes | Unclear | Interactive | No |
| 5 | T. L. Guldberg | 2011 | Patients | Unclear | Yes | Yes | Yes | Interactive | Yes |
| 6 | B. Guthrie | 2016 | Patients | Less than monthly | Yes | Yes | No | Static | Yes |
| 7 | W. Y. Lim | 2018 | Number of prescriptions | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 8 | J. A. Linder | 2010 | Providers | Monthly | Yes | Yes | Yes | Interactive | No |
| 9 | J. W. Mold | 2008 | Providers | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 10 | G. Ogedegbe | 2014 | Patient | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 11 | S. Ornstein | 2010 | Patients | Unclear | No | Unclear | Unclear | Static | Yes |
| 12 | G. A. Pape | 2011 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 13 | D. Peiris | 2015 | Patient-level data analysis | Unclear | Yes | Yes | Yes | Interactive | Yes (partially) |
| 14 | I. Urbiztondo | 2017 | Individual data at patient and GP level | Weekly | Yes | Unclear | Unclear | Static | Yes |
| 15 | D. Vinereanu | 2017 | Patients | Unclear | Unclear | Unclear | Unclear | Static | Yes |
| 16 | W. C. Wadland | 2007 | Providers | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 17 | N. Winslade | 2016 | Number of dispensings | Once | Yes | Yes | No | Static | Yes (partially) |
| 18 | M. Bahrami | 2003 | Patients | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 19 | P. Elouafkaoui, | 2016 | Providers | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 20 | J. S. Gerber | 2013 | Patients | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 21 | Y. Hayashino | 2015 | Patients | Monthly | Yes | Yes | Yes | Static | Yes |
| 22 | L. G. Hemkens | 2017 | Providers | Less than monthly | Yes | Yes | No | Static | No |
| 23 | T.A. Holt | 2017 | Patients | Unclear | Yes | No | Yes | Interactive | No |
| 24 | N. H. McAlister | 1986 | Patients | Unclear | Yes | Yes | Yes | Static | Yes |
| 25 | D. R. Murphy | 2015 | Patients | Monthly | Unclear | No | Unclear | Static | Yes |
| 26 | M. S. Patel | 2018 | Patients | Monthly | Yes | Yes | Unclear | Interactive | Yes |
| 27 | S. A. Nejad | 2016 | Providers | Less than monthly | Unclear | Unclear | Yes | Static | Yes |
| 28 | L. P. Svetkey | 2009 | Patients | Less than monthly | Yes | Yes | Unclear | Static | Yes (partially) |
| 29 | J. Trietsch | 2017 | Providers | Unclear | Yes | Yes | Yes | Static | No |
Feedback features
| First author | Year | Unit of analysis | Frequency of the feedback | Evidence-based feedback | Benchmarks | Low cognitive load | Electronic feedback | Effect of intervention | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | O. P. Almeida | 2012 | Patients | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 2 | A. J. Avery | 2012 | Patients | Less than monthly | Unclear | Unclear | Unclear | Static | Yes |
| 3 | B. Bonevski | 1999 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 4 | C. A. Estrada | 2011 | Patients | Less than monthly | Unclear | Yes | Unclear | Interactive | No |
| 5 | T. L. Guldberg | 2011 | Patients | Unclear | Yes | Yes | Yes | Interactive | Yes |
| 6 | B. Guthrie | 2016 | Patients | Less than monthly | Yes | Yes | No | Static | Yes |
| 7 | W. Y. Lim | 2018 | Number of prescriptions | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 8 | J. A. Linder | 2010 | Providers | Monthly | Yes | Yes | Yes | Interactive | No |
| 9 | J. W. Mold | 2008 | Providers | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 10 | G. Ogedegbe | 2014 | Patient | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 11 | S. Ornstein | 2010 | Patients | Unclear | No | Unclear | Unclear | Static | Yes |
| 12 | G. A. Pape | 2011 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 13 | D. Peiris | 2015 | Patient-level data analysis | Unclear | Yes | Yes | Yes | Interactive | Yes (partially) |
| 14 | I. Urbiztondo | 2017 | Individual data at patient and GP level | Weekly | Yes | Unclear | Unclear | Static | Yes |
| 15 | D. Vinereanu | 2017 | Patients | Unclear | Unclear | Unclear | Unclear | Static | Yes |
| 16 | W. C. Wadland | 2007 | Providers | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 17 | N. Winslade | 2016 | Number of dispensings | Once | Yes | Yes | No | Static | Yes (partially) |
| 18 | M. Bahrami | 2003 | Patients | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 19 | P. Elouafkaoui, | 2016 | Providers | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 20 | J. S. Gerber | 2013 | Patients | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 21 | Y. Hayashino | 2015 | Patients | Monthly | Yes | Yes | Yes | Static | Yes |
| 22 | L. G. Hemkens | 2017 | Providers | Less than monthly | Yes | Yes | No | Static | No |
| 23 | T.A. Holt | 2017 | Patients | Unclear | Yes | No | Yes | Interactive | No |
| 24 | N. H. McAlister | 1986 | Patients | Unclear | Yes | Yes | Yes | Static | Yes |
| 25 | D. R. Murphy | 2015 | Patients | Monthly | Unclear | No | Unclear | Static | Yes |
| 26 | M. S. Patel | 2018 | Patients | Monthly | Yes | Yes | Unclear | Interactive | Yes |
| 27 | S. A. Nejad | 2016 | Providers | Less than monthly | Unclear | Unclear | Yes | Static | Yes |
| 28 | L. P. Svetkey | 2009 | Patients | Less than monthly | Yes | Yes | Unclear | Static | Yes (partially) |
| 29 | J. Trietsch | 2017 | Providers | Unclear | Yes | Yes | Yes | Static | No |
| First author | Year | Unit of analysis | Frequency of the feedback | Evidence-based feedback | Benchmarks | Low cognitive load | Electronic feedback | Effect of intervention | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | O. P. Almeida | 2012 | Patients | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 2 | A. J. Avery | 2012 | Patients | Less than monthly | Unclear | Unclear | Unclear | Static | Yes |
| 3 | B. Bonevski | 1999 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 4 | C. A. Estrada | 2011 | Patients | Less than monthly | Unclear | Yes | Unclear | Interactive | No |
| 5 | T. L. Guldberg | 2011 | Patients | Unclear | Yes | Yes | Yes | Interactive | Yes |
| 6 | B. Guthrie | 2016 | Patients | Less than monthly | Yes | Yes | No | Static | Yes |
| 7 | W. Y. Lim | 2018 | Number of prescriptions | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 8 | J. A. Linder | 2010 | Providers | Monthly | Yes | Yes | Yes | Interactive | No |
| 9 | J. W. Mold | 2008 | Providers | Unclear | Unclear | Yes | Unclear | Static | Yes |
| 10 | G. Ogedegbe | 2014 | Patient | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 11 | S. Ornstein | 2010 | Patients | Unclear | No | Unclear | Unclear | Static | Yes |
| 12 | G. A. Pape | 2011 | Patients | Unclear | Yes | Yes | Unclear | Interactive | Yes |
| 13 | D. Peiris | 2015 | Patient-level data analysis | Unclear | Yes | Yes | Yes | Interactive | Yes (partially) |
| 14 | I. Urbiztondo | 2017 | Individual data at patient and GP level | Weekly | Yes | Unclear | Unclear | Static | Yes |
| 15 | D. Vinereanu | 2017 | Patients | Unclear | Unclear | Unclear | Unclear | Static | Yes |
| 16 | W. C. Wadland | 2007 | Providers | Less than monthly | Unclear | Yes | Yes | Static | Yes |
| 17 | N. Winslade | 2016 | Number of dispensings | Once | Yes | Yes | No | Static | Yes (partially) |
| 18 | M. Bahrami | 2003 | Patients | Less than monthly | Yes | Unclear | Unclear | Static | No |
| 19 | P. Elouafkaoui, | 2016 | Providers | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 20 | J. S. Gerber | 2013 | Patients | Less than monthly | Yes | Yes | Yes | Static | Yes |
| 21 | Y. Hayashino | 2015 | Patients | Monthly | Yes | Yes | Yes | Static | Yes |
| 22 | L. G. Hemkens | 2017 | Providers | Less than monthly | Yes | Yes | No | Static | No |
| 23 | T.A. Holt | 2017 | Patients | Unclear | Yes | No | Yes | Interactive | No |
| 24 | N. H. McAlister | 1986 | Patients | Unclear | Yes | Yes | Yes | Static | Yes |
| 25 | D. R. Murphy | 2015 | Patients | Monthly | Unclear | No | Unclear | Static | Yes |
| 26 | M. S. Patel | 2018 | Patients | Monthly | Yes | Yes | Unclear | Interactive | Yes |
| 27 | S. A. Nejad | 2016 | Providers | Less than monthly | Unclear | Unclear | Yes | Static | Yes |
| 28 | L. P. Svetkey | 2009 | Patients | Less than monthly | Yes | Yes | Unclear | Static | Yes (partially) |
| 29 | J. Trietsch | 2017 | Providers | Unclear | Yes | Yes | Yes | Static | No |
Strengths and limitations
To our knowledge, this is the first systematic review that investigated electronic A&F only in primary care. One of the strengths of this review is that our search was identical to that of the last Cochrane review. By replicating the search strings of the Cochrane review and screening abstracts and full texts based on our inclusion and exclusion criteria, this review had a solid basis. Furthermore, a broad definition of electronic feedback was used, thus reducing the risk of missing relevant articles.
Our review also has several limitations. Because our results showed high heterogeneity, meta-analysis was not feasible, and no generalizable data could be produced. Furthermore, for the calculation of our risk of bias summary, every form of bias was considered as a key domain, except for performance bias, which may have produced too severe an overall risk of bias evaluation.
Conclusion
This systematic review included 29 articles that examined electronic A&F interventions in primary care, and 76% of the interventions were found to be effective. Approximately 75% of the studies provided electronic feedback without the ability to interact with it. Despite that the design of the electronic A&F interventions varied widely, approximately 70% of the included studies used benchmarks and high-quality evidence in the content and delivery of the feedback. In almost half of the studies, the cognitive load of feedback was not reported. Our findings suggest electronic A&F is effective in primary care for different conditions such as medication safety and preventive medicine. Some of the benefits of electronic A&F include its scalability and the potential to be cost effective. The use of benchmarks as comparators and feedback based on high-quality evidence are widely used features of electronic feedback in primary care. However, other important features such as the cognitive load of feedback and the frequency of feedback provision are poorly described in the design of many electronic A&F intervention, indicating that a better description or implementation of these features is needed. Developing a framework or methodology for automated A&F interventions in primary care could be useful for future research.
Acknowledgements
S.V.D.B., B.V., G.G., R.H. and P.V. contributed to the design and conceptualization of the study. S.V.D.B. performed the search. S.V.D.B., D.S. and P.V. performed the screening, data extraction and risk of bias assessment. S.V.D.B., D.S., B.V., G.G., R.H. and P.V. reviewed and edited the manuscript. The authors would like to thank Dr Anne-Catherine Vanhove for her assistance with the search and Jacques Adriaansen for his assistance with the tables.
Funding
There was no funding for this systematic review.
Ethics and other permissions
Ethical approval was not required.
Data availability statement
The data extraction sheets containing all the articles in this review are available as supplemental material. No new data were generated in support of this review.
Appendices
Appendix 1: PRISMA checklist
Appendix 2: Inclusion and exclusion criteria
Appendix 3: Search strings and results
Appendix 4: Data extraction sheet continuous outcomes
Appendix 5: Data extraction sheet dichotomous outcomes
Appendix 6: Risk of bias evaluation
