Behavioral Nudges to Improve Audit and Feedback Report Opening Among Antibiotic Prescribers: A Randomized Controlled Trial

Abstract

Background

Peer comparison audit and feedback has demonstrated effectiveness in improving antibiotic prescribing practices, but only a minority of prescribers view their reports. We rigorously tested 3 behavioral nudging techniques delivered by email to improve report opening.

Methods

We conducted a pragmatic randomized controlled trial among Ontario long-term care prescribers enrolled in an ongoing peer comparison audit and feedback program which includes data on their antibiotic prescribing patterns. Physicians were randomized to 1 of 8 possible sequences of intervention/control allocation to 3 different behavioral email nudges: a social peer comparison nudge (January 2020), a maintenance of professional certification incentive nudge (October 2020), and a prior participation nudge (January 2021). The primary outcome was feedback report opening; the primary analysis pooled the effects of all 3 nudging interventions.

Results

The trial included 421 physicians caring for >28 000 residents at 450 facilities. In the pooled analysis, physicians opened only 29.6% of intervention and 23.9% of control reports (odds ratio [OR], 1.51 [95% confidence interval {CI}, 1.10–2.07], P = .011); this difference remained significant after accounting for physician characteristics and clustering (adjusted OR [aOR], 1.74 [95% CI, 1.24–2.45], P = .0014). Of individual nudging techniques, the prior participation nudge was associated with a significant increase in report opening (OR, 1.62 [95% CI, 1.06–2.47], P = .026; aOR, 2.16 [95% CI, 1.33–3.50], P = .0018). In the pooled analysis, nudges were also associated with accessing more report pages (aOR, 1.28 [95% CI, 1.14–1.43], P < .001).

Conclusions

Enhanced nudging strategies modestly improved report opening, but more work is needed to optimize physician engagement with audit and feedback.

Clinical Trials Registration

NCT04187742.

Antibiotic prescribing is challenging in long-term care (LTC) facilities. About half of antibiotic prescriptions to LTC residents are unnecessary or inappropriate [1–3]. Total antibiotic use varies 10-fold across LTC facilities, and residents of facilities with high antibiotic use experience more antibiotic-related harms than those residing in low-use facilities [4]. This variability in antibiotic use is driven by prescriber habits, rather than resident characteristics [5]. Prescribers exhibit sustained personal preferences for initiating antibiotics, selecting specific agents, and using prolonged durations of treatments; these differences are not explained by resident case mix [5].

Antimicrobial stewardship can be effective in reducing antibiotic overuse in LTC facilities, but most interventions are labor intensive with unclear sustainability [6]. However, we have demonstrated that using peer comparison audit and feedback to make physicians aware of their personal prescribing habits is a feasible and scalable approach to improve antimicrobial prescribing in LTC [7]. In an analysis of 1238 physicians caring for 96 185 LTC residents, we demonstrated that audit and feedback was associated with a modest reduction in use of prolonged-duration antibiotics. This translated into annual reductions of >300 000 days of antibiotic use in Ontario LTC facilities [7]. An embedded randomized controlled trial (RCT) did not detect a difference between using a novel, dynamic dashboard or a traditional, static paginated report. The interactive and user-friendly dashboard has now been introduced into permanent use, with ongoing regular reporting of antibiotic initiation and duration indicators to participating physicians. However, a persistent challenge to electronically disseminated audit and feedback reports is finding ways to encourage as many physicians as possible to open the report and view their own data. For example, in a prior iteration of these reports focused on antipsychotic prescribing, only a minority (14%) of physicians who were enrolled to receive the report actually viewed their feedback [8].

Therefore, the next incremental goal in this implementation laboratory was to seek improvements in the rates of report opening of the dynamic dashboard via modifications to the notification email [9]. Insights from behavioral sciences can be used to influence not only antimicrobial prescribing, but also a variety of other clinician behaviors, including their likelihood of opening prescribing reports in the first place. Nudging is a concept aimed at guiding human behavior while preserving autonomy and is rapidly emerging in a range of fields including public policy, finance, energy consumption, and health [9–12]. In 3 sequential email notifications, we randomized prescribers to a traditional email notification or a novel behavioral nudge.

METHODS

General Study Design and Setting

We conducted a randomized trial (ClinicalTrials.gov identifier NCT04187742) among Ontario LTC physicians who had signed up for a voluntary ongoing report providing peer comparison audit and feedback on their antibiotic prescribing practices. Physicians were notified about the availability of an updated report in January 2020, October 2020, and January 2021; physicians were randomized to receive either a traditional email notification, or a novel email with a behavioral nudge to promote report opening. The primary objective of the study was to determine if the nudging interventions were associated with increased report opening; the primary analysis was pooled across all 3 behavioral nudges. The study was approved by the Research Ethics Board (REB) of Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada (REB project identification number 441-2017).

Participating Physicians and the Feedback Report

Ontario Health (an agency created by the Government of Ontario with a mandate to connect and coordinate the province’s healthcare) launched a voluntary audit and feedback report (MyPractice) in 2015 to provide LTC prescribers with quarterly information on their antipsychotic and other neurotropic medication prescribing in relation to their peers [13, 14]. In 2019, antibiotic indicators were added to the report, including (1) the percentage of residents initiated on an antibiotic and (2) the percentage of antibiotic prescriptions with duration exceeding 7 days [7]. Prescribers sign up for this report on a voluntary basis and provide their preferred email address. At each new report release with updated data, physicians receive an email notification with an embedded link to a portal where they can log in to access their personalized report. This study involved randomization to a novel behavioral nudge planned for each of the first 3 quarterly email notification updates in 2020; due to the severe acute respiratory syndrome coronavirus 2 pandemic wave 1, the second email intervention was postponed until October 2020, and due to pandemic wave 2, the third email intervention was postponed until January 2021. Participating physicians were eligible for each of the 3 randomized email interventions if they had cared for at least 6 Ontario LTC residents in the 3 months preceding the email.

To be most effective, audit and feedback should be simple, timely, repeated, regular, and transmitted from an identified and respected source and should contain a clear, actionable message with thoughtful comparisons [15]. Our audit and feedback report itself was developed with input from infectious diseases, implementation science, information technology, and quality improvement specialists, and then improved through an iterative, user-centered design process as previously described [7]. The report includes antibiotic duration and initiation indicators for the individual physician, with a comparison to Ontario LTC prescriber peers, explanations of the data sources, guides to interpretation, systematic prescribing change ideas, and also simple prescribing change ideas that can be accomplished immediately [7]. The 3 email interventions in this RCT were generated by the same multidisciplinary research team, and then subjected to pilot and sensibility testing on LTC prescribers.

Randomization, Allocation Concealment, and Blinding

Physicians already enrolled in the MyPractice program prior to January 2020 were randomized up-front to intervention or control groups for each of the first, second, and third intervention emails. Randomization was generated by an analyst unaware of individual physician identities, and was conducted in blocks of 8 so that there were similar numbers of physicians randomized to each possible combination sequence of intervention and control email allocation (ie, intervention-intervention-intervention, intervention-intervention-control, intervention-control-intervention, control-intervention-intervention, control-control-intervention, control-intervention-control, intervention-control-control, control-control-control). Additional randomization slots were generated in anticipation that some new physicians might join the reporting system prior to the second and/or third email intervention. The Ontario Health team was aware of intervention groups after allocation, so that they could transmit the correct emails to each participant; the physicians were, by definition, aware of the email to which they were exposed but were not notified that the email characteristics had been altered. The study team was blinded to intervention group, and all analyses were conducted prior to unblinding the assignments.

Social Peer Comparison Nudge

The first intervention email used elements of social peer comparison to draw physicians into the report [16, 17], with the title “How do your prescription lengths compare to other LTC physicians?” The email text provided a performance benchmark that “1 in 4 LTC physicians have minimized their use of longer duration prescriptions to less than 12% of their antibiotic prescriptions” and then prompted physicians to “Click to see how you compare” (Supplementary Figure 1A).

Maintenance of Professional Certification Incentive Nudge

The second intervention email aimed to improve report opening by reminding physicians that they get a secondary personal benefit by using the report—namely, credits toward their annual maintenance of professional certification [18]. The Canadian College of Family Physicians requires that physicians achieve a minimum of 25 “Mainpro+” credits per year toward professional certification; the feedback report is an accredited activity that is valued at 5 credits per use. The nudging email title reminded prescribers to “Earn Mainpro+ certified credits” and the ensuing text explained the number of credits and process for achieving credits (Supplementary Figure 1B).

Prior Participation Nudge

The third intervention email reminded prescribers whether they had opened at least 1 of their previous 2 reports. The title stated, “Many doctors have viewed their MyPractice report” [19]. Email text was then stratified based on prior opening behavior to read either “You are among the LTC doctors who have viewed their MyPractice report at least once in 2020” (Supplementary Figure 1C) or “We noticed that you have not yet had a chance to see your report” (Supplementary Figure 1D). The traditional comparator email was kept unchanged across the 3 trials (Supplementary Figure 1E).

Primary Outcome

The prespecified primary study outcome was whether the prescribers opened their report within 30 days of receipt of the email notification.

Secondary Outcomes

Secondary process outcomes of interest were the percentage of prescribers who opened the notification email itself, rather than the report link contained within the email, and the number of pages viewed in the dashboard report. Secondary clinical outcomes included antibiotic initiation (percentage of residents prescribed an antibiotic in the 3 months following the email notification) and antibiotic duration (percentage of antibiotic treatment courses exceeding 7 days in the 3 months following the email notification).

Data Sources

The primary and secondary report use outcomes were extracted as routinely available metrics from the dynamic dashboard information system (Dundas BI, Inc). Physicians were linked by their unique encoded prescriber number to ICES administrative databases. ICES is an independent, nonprofit research institute whose legal status under Ontario’s health information privacy law allows it to collect and analyze healthcare and demographic data, without consent, for health system evaluation and improvement. The linked administrative databases at ICES that capture these data have been well validated [20, 21] and used for extensive related research involving antibiotic use [22, 23], quality of LTC [24], and antibiotic use in LTC facilities [4, 5]. At ICES, the antibiotic prescribing data were obtained from the Ontario Drug Benefits database, which includes information on all antibiotics prescribed to Ontarians aged ≥65 years or living in LTC facilities, with an accuracy exceeding 99% [25]; The Continuing Care Reporting System Resident Assessment Instrument Minimal Dataset Version 2.0 was used to determine resident characteristics; the ICES physician database was used to determine prescriber characteristics. These datasets were linked using unique encoded identifiers and analyzed at ICES.

Statistical Analysis

The study was analyzed in an intention-to-treat fashion as per Consolidated Standards of Reporting Trials (CONSORT) reporting guidelines (Supplementary Figure 2) [26]. In the primary analysis we pooled the results of all 3 intervention emails (vs control), and in secondary analyses we examined the results of each intervention email separately. The report opening metric was compared between physicians receiving intervention vs control emails, using logistic regression with a covariate for intervention vs control, a categorical variable for each 3-month time period, and a random effect for physician. An additional adjusted analysis was performed in which we accounted for physician age, sex, medical school graduation outside of Canada, number of years of practice, number of patients, baseline antibiotic initiation percentage in 2018, and baseline antibiotic prolongation beyond 7 days in 2018. Similar logistic regression analyses were performed for email opening, while Poisson regression was used for the count outcome of number of report pages accessed. Logistic models evaluating impact on antimicrobial initiation and duration outcomes also adjusted for resident characteristics from their most recent annual LTC assessment including age, sex, 14 comorbidities, degree of functional dependence for each activity of daily living, bowel/bladder incontinence, and hearing/visual impairment [4, 27]. For the prior report opening intervention, we conducted a stratified analysis to determine impact in those who had opened or not opened prior reports. For the overall pooled analysis, we examined prescriber subgroups including male vs female physicians, younger vs older physicians, and low vs high prescribers based on 2018 baseline data. Statistical analyses were conducted using SAS Enterprise Guide 9.4 software.

Power Calculation

Based on prior physician report, login rates of approximately 33%, and between-physician standard deviation of 10%, we determined that randomizing 350 physicians would enable us to detect an odds ratio (OR) of 1.35 for report opening associated with the intervention vs comparator emails (power 0.80, 2-tailed α = .05).

RESULTS

Prescriber Characteristics

A total of 421 enrolled LTC physicians met eligibility criteria for randomization to intervention nudge or control emails. In January 2020, 211 received the social peer comparison intervention nudge and 210 the control email; in October 2020, 212 received the maintenance of professional certification incentive nudge and 209 the control email; in January 2021, 209 received the prior report opening nudge and 212 the control email (Supplementary Figure 2).

The majority of physicians were male (65.1%), their median age was 58 years (interquartile range [IQR], 49–66 years), and they had been in practice for a median of 32 years (IQR, 20–40 years) and cared for 46 (IQR, 11–99) LTC residents. Physician characteristics were well balanced across the intervention and control arms in the first intervention (Table 1), as well as in the subsequent email interventions (Supplementary Tables 1 and 2).

Resident Characteristics

The enrolled LTC physicians collectively cared for >28 154 residents at 450 facilities. The residents were a median age of 86 years (IQR, 78–91 years), predominantly female (67.8%), and with high prevalence of dementia (64.5%) and other comorbidities. The resident characteristics were well balanced across the intervention and control arms in the first email nudging intervention (Table 1), as well as in the subsequent email nudging interventions (Supplementary Tables 1 and 2).

Primary Outcome: Report Opening

Overall, 249 (59.1%) physicians opened 0 reports, 58 (13.8%) opened 1 report, 62 (14.7%) opened 2 reports, and 52 (12.4%) opened all 3 reports. In the pooled analysis, physicians only opened 29.6% of intervention reports and 23.9% of control reports, translating to an unadjusted OR of opening the intervention report of 1.51 (95% confidence interval [CI], 1.10–2.07; P = .011) (Figure 1, Table 1). After adjustment, the pooled interventions remained associated with increased report opening (adjusted OR [aOR], 1.74 [95% CI, 1.24–2.45], P = .0014) (Table 2).

The social peer comparison email intervention was not significantly associated with report opening (OR, 1.45 [95% CI, .92–2.28], P = .113; aOR, 1.49 [95% CI, .91–2.44], P = .111). The maintenance of certification incentive email intervention was also not associated with a detectable impact on report opening (OR, 1.03 [95% CI, .67–1.58], P = .898; aOR, 1.30 [95% CI, .81–2.10], P = .274). The prior report-opening email intervention was associated with an increase in report opening in both unadjusted (OR, 1.62 [95% CI, 1.06–2.47], P = .026) and adjusted (aOR, 2.16 [95% CI, 1.33–3.50], P = .0018) analyses (Table 2). The benefit in the prior report-opening email was seen particularly in those who had previously opened their reports (aOR, 2.38 [95% CI, 1.40–4.10], P = .001), and less so in those who had not opened either of their previous reports (aOR, 1.67 [95% CI, .36–7.82], P = .51).

Secondary Outcomes and Subgroup Analyses

In the pooled analysis, intervention vs control emails were significantly associated with accessing more report pages (mean, 1.70 ± 2.99 pages vs 1.31 ± 2.67 pages; adjusted relative risk [aRR], 1.28 [95% CI, 1.14–1.43], P < .001). As expected, there was no impact on email opening in the pooled intervention vs control emails (57.4% vs 55.8%; aRR, 1.10 [95% CI, .78–1.53], P = .595). Antibiotic initiation and duration were not significantly different across the intervention and control groups (Table 3). In subgroup analyses, we did not detect a statistically significant effect modification by physician sex, age, or baseline prescribing rate.

DISCUSSION

Peer comparison audit and feedback reports are a powerful method to improve physician prescribing practices and are scalable over large health systems when distributed electronically [7, 16]. However, these feedback systems can only be effective if physicians actually open and view their reports. In this RCT involving audit and feedback reporting to 421 physicians prescribing to >28 000 LTC residents across 450 facilities, we have demonstrated that enhanced email nudging strategies can modestly improve report opening. The effect size is in keeping with a recent mega-study of nudges to improve influenza vaccine uptake [28]. In our study, a prior report-opening nudge had the greatest impact on report opening; a social peer comparison nudge had a similar magnitude of association but the change in report opening was not statistically significant; a reminder about an incentive based on credits toward maintenance of professional certification did not improve report-opening rates. However, less than one-third of physicians opened any of the individual emailed reports in this study, and there were no detectable improvements in antimicrobial prescribing, suggesting that there is much work yet to be done to increase physician engagement with audit and feedback.

Reminding prescribers that the MyPractice report distributors are aware who has opened their prior reports was effective in enhancing report opening, and this benefit was most evident in those who had previously opened their reports. These findings are in line with prior experiments by the American Red Cross, which indicated that reminding people that they are a previous blood donor increases their likelihood of a subsequent donation [19]. An individual’s “identity” or “sense of self” may play a significant role in their behavior [29]. If a person self-identifies as generous and charitable, reminding them of this facet of their identity may help nudge toward further charitable acts. Similarly, if a clinician identifies themselves as somebody interested in performance improvement, then reminding them that they opened prior reports may encourage them to open their next feedback report. Our finding is also consistent with prior audit and feedback literature, which suggests that positive framing is more effective than negative framing [30].

The peer social comparison nudge used a similar approach to the audit and feedback report itself, to coax prescribers to open the report. Multiple studies, including RCTs among antimicrobial prescribers, have indicated that providing peer comparisons can improve prescribing practices [16]. Our study indicates that the same concept may help encourage physicians to open the report itself, given that the magnitude of effect was similar to that seen with the prior report-opening nudge. The intervention email text emphasized an achievable benchmark already attained by “1 in 4 LTC physicians,” and previous work has indicated that this can be an effective motivator for physicians [17]. However, there was no statistically significant improvement with this intervention, and so the potential for impact on report opening requires further study.

Reminding physicians that participation entitles them to maintenance of certification credits was not associated with an improvement in report opening. Behavioral change interventions that indicate a personal benefit can help nudge the desired behavior by highlighting “what’s in it for me” [31, 32]. The concept of this intervention was to incentivize clinicians with an additional personal benefit beyond the direct benefits of engaging with their prescribing data. Maintenance of certification credits have been used elsewhere to encourage participation in quality improvement activities [18, 33], but the impact of this technique has not been rigorously studied. It is important to note, though, that our study did not test whether addition of maintenance of certification credits improved report opening; the 5-credit opportunity was already available to prescribers, and the intervention email merely provided added emphasis or reminder of this side benefit of feedback report viewing. Perhaps physicians were already well aware of this maintenance of certification opportunity.

Our study was limited to physicians who had voluntarily enrolled in the audit and feedback program, and so it is not clear if we can generalize findings to prescribers who do not seek involvement in such quality improvement initiatives; their opening rates would be expected to be even lower at baseline. Our primary outcome established that the behavioral nudges improved report opening, but that does not necessarily mean it will translate into action in the form of improved prescribing. However, our prior province-wide analysis did indicate that presenting these antibiotic indicators was associated with substantial improvements in antibiotic use [7]. The current study, though, was conducted during the coronavirus disease 2019 pandemic, during which there were many additional pressures on LTC antibiotic prescribing behaviors and related changes in prescribing practice [34]. The primary outcome of report opening was likely reduced by COVID-19 time pressures on clinicians, even though we delayed email transmissions to avoid peak pandemic waves in Ontario LTC facilities. Nonetheless, the effect should be nondifferential among those randomized to control vs intervention emails. Our study strengths included a population-wide intervention involving large numbers of prescribers and high-risk patients and, most important, a rigorous, RCT design. Comprehensive evaluations to inform iterative improvements of quality improvement strategies can provide an ideal contribution to learning health systems [9].

CONCLUSIONS

In the context of an RCT, we have demonstrated that emails incorporating behavioral nudges can be a potentially effective way to improve audit and feedback report opening. Therefore, nudging techniques should be delivered beyond the feedback reports and permeate into the notification email. One potentially helpful approach is to extend social comparison and benchmark messaging beyond the report itself as this can help draw people into the data to learn if they have achieved target benchmarks. The most helpful approach is to compliment people on opening previous iterations of a feedback report, thereby reinforcing their identity as self-improvers. We can target feedback to those who previously open reports and remind them about their prior opening patterns. However, low overall opening rates, and lack of detectable improvement in antibiotic prescribing, suggest that more behavioral science research is needed to further encourage clinicians to open their feedback reports [15, 30]. In particular, we must learn ways to target the nonopeners who do not already engage with their reports, such that they will have an opportunity to interact with the data and optimize their antimicrobial prescribing practices.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Disclaimer. Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. The study acknowledges a terrific collaboration between Public Health Ontario, ICES and Ontario Health.

Financial support. This work was supported by a collaboration across Public Health Ontario, Ontario Health, and ICES. ICES is funded by an annual grant from the Ontario Ministry of Health and the Ministry of Long-Term Care. This study also received funding from the Canadian Institutes for Health Research (grant number 378064 to N. D.). N. I. receives salary support in the form of a Canada Research Chair as well as a clinician scholar award from the Department of Family Community Medicine at the University of Toronto.

Potential conflicts of interest. All authors: No reported conflicts of interest.

All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

References