Abstract

Appalachian regions of Kentucky and Ohio are hotspots for colorectal cancer (CRC) mortality in the USA. Screening reduces CRC incidence and mortality; however, screening uptake is needed, especially in these underserved geographic areas. Implementation science offers strategies to address this challenge. The aim of the current study was to conduct multi-site, transdisciplinary research to evaluate and improve CRC screening processes using implementation science strategies. The study consists of two phases (Planning and Implementation). In the Planning Phase, a multilevel assessment of 12 health centers (HC) (one HC from each of the 12 Appalachian counties) was conducted by interviewing key informants, creating community profiles, identifying HC and community champions, and performing HC data inventories. Two designated pilot HCs chose CRC evidence-based interventions to adapt and implement at each level (i.e., patient, provider, HC, and community) with evaluation relative to two matched control HCs. During the Implementation Phase, study staff will repeat the rollout process in HC and community settings in a randomized, staggered fashion in the remaining eight counties/HCs. Evaluation will include analyses of electronic health record data and provider and county surveys. Rural HCs have been reluctant to participate in research because of concerns about capacity; however, this project should demonstrate that research does not need to be burdensome and can adapt to local needs and HC abilities. If effective, this approach could be disseminated to HC and community partners throughout Appalachia to encourage the uptake of effective interventions to reduce the burden of CRC.

Lay Summary

We conducted a multi-site study to evaluate and improve CRC screening processes using implementation science strategies at multiple levels including the patient, provider, health center, and community. Our goals were to increase rates of guideline-recommended CRC screening, follow-up, and referral-to-care in an Appalachian, medically underserved population.

Implications

Practice: Results from this project could be significant to researchers and practitioners working in rural and underserved areas where individualized implementation and sustainability of evidence-based interventions related to colorectal screening at multiple levels are necessary.

Policy: Policymakers should explore ways to support evidence-based interventions in health centers such as requirements that electronic health records help providers identify patients who are not up to date with colorectal cancer screening and follow-up with abnormal results.

Research: Future research should explore whether this implementation approach could be disseminated to other Appalachian areas.

INTRODUCTION

The 13-state Appalachian region of the United States (U.S.) is a recognized “hotspot” for colorectal cancer (CRC), with higher than average incidence and mortality rates [1]. The Appalachian regions of the two states participating in this multi-site project, Kentucky (KY) and Ohio (OH), experience significant socioeconomic disparities [2, 3] and some of the highest incidence and mortality rates for CRC in the nation [4], reinforcing the need to reduce the disease burden in both states. Coupled with increased CRC incidence and mortality, CRC screening prevalence is lower in Appalachian regions of KY and OH as compared to non-Appalachian regions [5]. In 2019 (at the start of this project), the U.S. Preventive Services Task Force recommended timely CRC screening in average-risk individuals aged 50–75 to reduce CRC incidence and mortality [6].

In general, Appalachian residents have higher poverty and unemployment rates, lower educational attainment and income, and poorer health than non-Appalachian residents [2, 7]. This project focuses on six Appalachian counties in KY and OH (12 total), selected due to high rates of CRC incidence and mortality and lower rates of CRC screening compared to national rates [4]. All counties are designated by the Health Resources and Services Administration as Health Professional Shortage Areas due to high poverty. Both stool-based testing (e.g., fecal immunochemical test [FIT], FIT-DNA, etc.) and colonoscopy screening are priorities to ensure access to guideline-recommended screening tests in all counties.

Previous experience with Appalachian-based cancer control initiatives helped identify key concepts to guide this project. First, while providers and health care systems have clear roles in recommending CRC screening and providing access, screening rates are also related to patient factors such as knowledge and possible fear of screening and CRC [2, 8, 9]. Second, characteristics and life circumstances of Appalachians are important considerations in selecting intervention strategies. For instance, Appalachians often live in tightly knit, rural communities in which information from “outsiders” is scrutinized to determine if it can be trusted [10] To engage the population, healthcare providers must disseminate information via methods that reflect local values, interests, and communication styles [11].

While intrapersonal barriers (i.e., embarrassment, fear, worry) and health care provider-related factors contribute to low CRC screening rates in Appalachia, screening rates are also influenced by community factors such as poverty, community norms, culture, and access to health care [12–14]. Intervening on multiple levels of influence may yield greater impact for health disparity reduction initiatives [15], and paying particular attention to both upstream and downstream factors, as highlighted in the Warnecke Model [16], can improve understanding of not only the multilevel factors that hinder and facilitate adherence to CRC screening recommendations, but also the relationships between levels. Although safety net clinics and providers remain critical sources of preventive care in rural Appalachia, community partners (e.g., churches, health and wellness centers, community health coalitions) also play a significant role in providing CRC education to facilitate changing residents’ norms about the disease, screening, and follow-up care.

Appalachian populations face not only personal and community barriers to screening but also provider and health center (HC) level barriers. Patient-level barriers to CRC screening commonly cited among Appalachian populations include low perceived risk of CRC; lack of awareness about the perceived benefits of screening; poor knowledge about screening options and guidelines; provider distrust; negative perceptions of stool-collection procedures and preparation for colonoscopy; privacy issues; lack of public transportation and poor safety of rural roads; shortage of specialists; and associated costs (the procedure, lack of health insurance, time off work for the patient and transport person) [8, 9, 17–20].

One of the most frequently cited reasons for receiving a CRC screening is a physician recommendation [17]. Provider-level challenges to recommending CRC screening include limited time, misperception of patients’ willingness to complete a CRC screening test, poor communication skills, high patient volume, acute medical concerns, and procedural and reimbursement issues [9, 13, 17]. At the HC-level, various barriers to receipt of CRC screening have been identified, including lack of a written, consistent plan promoting CRC screening and follow-up, as well as provider and HC availability within the region. For example, most safety net HCs and systems within Appalachia OH and KY do not provide endoscopy services, meaning that referrals and additional consultation visits need to take place before screening can occur, and patients may need to travel long distances to receive endoscopy services [18]. Moreover, complications associated with insurance reimbursements have led to some HC staff reportedly delaying screenings until Medicaid or Medicare agree to procedure costs [18]. This paper describes the protocol of an implementation science study involving two phases, a pilot phase and implementation phase, to evaluate CRC screening strategies and improve CRC screening rates in Appalachian KY and OH.

METHODS

This study is being conducted as part of the National Cancer Institute’s (NCI)-funded Cancer MoonshotSM Initiative, the Accelerating CRC Screening and Follow-up through Implementation Science (ACCSIS) Program. The aim of ACCSIS is to conduct multi-site, coordinated, transdisciplinary research to evaluate and improve CRC screening processes using implementation science strategies. As one of five funded initiatives, the overall goals of this two-phase project are to (i) ­contribute to the evidence base for a multilevel intervention (MLI) (focused on patient, provider, health center and community components) that increase rates of guideline-recommended CRC screening, follow-up, and referral-to-care, particularly in rural, medically underserved populations, and (ii) help showcase best practices for how MLIs could be scaled-up to reduce the burden of CRC in the USA. This protocol was approved by University Institutional Review Board (IRB) (#2018C0166) and registered at clinicaltrials.gov (NCT04427527).

Specific aims for the 1-year Planning-Exploratory Phase (Phase 1) were to: (i) pilot test, measure, and refine a MLI implementation process to increase rates of CRC screening, follow-up, and referral-to-care in two counties: one in Appalachian KY and one in Appalachian OH; and (ii) provide evidence supporting transition to the Implementation Phase. Phase 1 results have been reported previously [21]. Specific aims for the remaining four years—the Implementation Phase (Phase 2)—include (i) testing the revised, ­customizable MLI implementation process in a group randomized, delayed intervention design to assess the impact of a MLI on increasing rates of CRC screening, follow-up, and referral-to-care in four Appalachian KY and four Appalachian OH counties; and (ii) assessing dissemination and sustainability of the intervention.

Study framework

The MLI will be evaluated using a group randomized trial (GRT) [22–24]. The eight non-pilot counties were assigned to one of two study groups (early vs. delayed), and outcome measures (rates of CRC screening and follow-up) will be obtained from HC-level electronic health record (EHR) data and a county-level behavioral assessment telephone survey like the Center for Disease Control and Prevention’s (CDC’s) Behavioral Risk Factor Surveillance System (BRFSS). Study group was assigned at the county-level to avoid contamination as residents, patients, and providers may visit or practice in more than one HC within a network.

Study design

The intervention and evaluation are being introduced in a staggered fashion to address logistical issues that would arise if multiple counties within a state were to begin the intervention simultaneously (Fig. 1). Clinics in counties were selected based on location in Appalachian KY and OH counties with disparate CRC rates and not overburdened at the time with other research initiatives. Counties were paired according to state and number of patients seen at the participating HC in each county. Pair 1 consisted of the pilot counties and two matched control (delayed intervention) counties. The order of intervention rollout for the other two pairs was randomly determined. Study group (early vs. delayed intervention) were also randomly assigned within Pairs 2 and 3. Thus, eight counties were randomized to a study group and four counties were not randomized. Pairing each intervention county with a delayed intervention county controlled for secular trends in CRC screening rates. The simultaneous introduction of the intervention in one KY and one OH county control for state differences over time. The county assigned to the early intervention group receives the MLI at the beginning of Phase 2 and the other county in the pair (Fig. 1) receives the program 12 months later (delayed group). At baseline as well as at 12 (end of Phase 1), 24 (end of Active Implementation in the Early Intervention group), 36, and 48 months, HC-level, and county-level CRC screening rates will be obtained using the EHRs and random survey samples from each county to assess trends. Telephone surveys of the random sample of county residents, age 50–74, are being conducted by trained interviewers.

Study design: data collection time points by county pairs and timeline.
Fig 1

Study design: data collection time points by county pairs and timeline.

Table 1 shows potential interventions to address patient-level, provider-level, health center, and community level barriers, which were found through a search of Evidence-Based Cancer Control Programs and a review of the literature [25]. Directors in each state managed intervention selection, which was discussed during state- and program-level planning meetings and tracked via detailed meeting notes. Clinics and counties could change their selections over time if needed, and such changes were documented as adaptations in meeting notes. For example, patients could receive printed CRC and CRC screening information to address patient-level barriers [26–34]. Providers could receive training in making strong recommendations to increase the uptake of CRC screening [29, 35]. To address barriers at the HC-level, EHR modifications could occur to improve the tracking of CRC screening [36]. This study engages local community volunteers to provide education about CRC and CRC screening and to advocate as community champions to change community norms. Community Advisory Boards (CAB) were formed in both states from these community and HC champions to review aspects of the study and advise study investigators. The respective CABs consist of about 10 members each and meets two times a year.

Table 1

Priority multilevel intervention strategies to increase colorectal cancer screening and follow-up

PatientProviderHealth centreCommunity
• Telephone counselling—screening and appointment reminders and barriers counselling from patient navigator [32, 37]
• Patient education via posters, brochures, table tents and/or social media [32, 35, 36]
• CRC risk assessment to assess patient CRC risk
• Mailed FIT kits w/ screening recommendation letter and printed education material [32, 38]
• Provider education/ CME for clinic staff to increase CRC and CRC screening knowledge among health care teams [33, 34]
• Academic detailing for physicians to improve complete diagnostic evaluation recommendation and performance [28, 33]
• Provider assessment w/ feedback to increase knowledge of individual- and/or team-level CRC screening recommendations, completion and referrals [33]
• Electronic health record prompts to physicians for CRC screening and follow-up reminders
• Patient navigation—reminder calls w/barriers counselling [32]
• Appointment reminders (cards, phone calls, texts or emails) [37]
• Written CRC screening and follow-up pathway
• Education and awareness seminars in local settings (e.g., health department, health fair, library, church)
• Use of inflatable colon at community events [31]
• Distribution of FIT kits
• Advocacy efforts with support from local coalitions and government
PatientProviderHealth centreCommunity
• Telephone counselling—screening and appointment reminders and barriers counselling from patient navigator [32, 37]
• Patient education via posters, brochures, table tents and/or social media [32, 35, 36]
• CRC risk assessment to assess patient CRC risk
• Mailed FIT kits w/ screening recommendation letter and printed education material [32, 38]
• Provider education/ CME for clinic staff to increase CRC and CRC screening knowledge among health care teams [33, 34]
• Academic detailing for physicians to improve complete diagnostic evaluation recommendation and performance [28, 33]
• Provider assessment w/ feedback to increase knowledge of individual- and/or team-level CRC screening recommendations, completion and referrals [33]
• Electronic health record prompts to physicians for CRC screening and follow-up reminders
• Patient navigation—reminder calls w/barriers counselling [32]
• Appointment reminders (cards, phone calls, texts or emails) [37]
• Written CRC screening and follow-up pathway
• Education and awareness seminars in local settings (e.g., health department, health fair, library, church)
• Use of inflatable colon at community events [31]
• Distribution of FIT kits
• Advocacy efforts with support from local coalitions and government
Table 1

Priority multilevel intervention strategies to increase colorectal cancer screening and follow-up

PatientProviderHealth centreCommunity
• Telephone counselling—screening and appointment reminders and barriers counselling from patient navigator [32, 37]
• Patient education via posters, brochures, table tents and/or social media [32, 35, 36]
• CRC risk assessment to assess patient CRC risk
• Mailed FIT kits w/ screening recommendation letter and printed education material [32, 38]
• Provider education/ CME for clinic staff to increase CRC and CRC screening knowledge among health care teams [33, 34]
• Academic detailing for physicians to improve complete diagnostic evaluation recommendation and performance [28, 33]
• Provider assessment w/ feedback to increase knowledge of individual- and/or team-level CRC screening recommendations, completion and referrals [33]
• Electronic health record prompts to physicians for CRC screening and follow-up reminders
• Patient navigation—reminder calls w/barriers counselling [32]
• Appointment reminders (cards, phone calls, texts or emails) [37]
• Written CRC screening and follow-up pathway
• Education and awareness seminars in local settings (e.g., health department, health fair, library, church)
• Use of inflatable colon at community events [31]
• Distribution of FIT kits
• Advocacy efforts with support from local coalitions and government
PatientProviderHealth centreCommunity
• Telephone counselling—screening and appointment reminders and barriers counselling from patient navigator [32, 37]
• Patient education via posters, brochures, table tents and/or social media [32, 35, 36]
• CRC risk assessment to assess patient CRC risk
• Mailed FIT kits w/ screening recommendation letter and printed education material [32, 38]
• Provider education/ CME for clinic staff to increase CRC and CRC screening knowledge among health care teams [33, 34]
• Academic detailing for physicians to improve complete diagnostic evaluation recommendation and performance [28, 33]
• Provider assessment w/ feedback to increase knowledge of individual- and/or team-level CRC screening recommendations, completion and referrals [33]
• Electronic health record prompts to physicians for CRC screening and follow-up reminders
• Patient navigation—reminder calls w/barriers counselling [32]
• Appointment reminders (cards, phone calls, texts or emails) [37]
• Written CRC screening and follow-up pathway
• Education and awareness seminars in local settings (e.g., health department, health fair, library, church)
• Use of inflatable colon at community events [31]
• Distribution of FIT kits
• Advocacy efforts with support from local coalitions and government

Process: how it will happen

Phase 1 (planning–exploratory)

During the planning–exploratory phase, the potential for implementing and evaluating a multilevel assessment of all 12 counties was assessed with healthcare provider and community partners and the CAB by conducting key informant interviews, creating community profiles, identifying healthcare provider and community champions, and performing HC data inventories. The assessment captured data about CRC burden and annual CRC screening rates, community, and patient socio-demographic profiles, and available community, public health, health care, and social services resources and partnerships. Additional information pertaining to key informant interviews, the CAB and HC inventories have been published elsewhere [21].

Healthcare provider and community champions

During Phase 1, community, and healthcare provider champions were identified through key informant interviews and trained to lead and advise the implementation of the adapted MLI in the respective sites and participate on the CAB. Ideal characteristics of champions included having an in-depth knowledge of the community or as a member of HC leadership, leadership qualities, and participation in health promotion and prevention programs. Additionally, insight into issues that were likely to affect intervention implementation, such as time or cost, were desirable attributes. Champions were trained by reviewing study goals and relating them to the champions’ locale with the University project managers from KY and OH. The training of community champions emphasized interactions with community groups and individuals to facilitate recruitment to local events that often used the “Inflatable Colon” [33]. For HC champions, the training focused on integrating the study goals into the HC workflow.

Pilot rollout

To ensure that the iterative evidence-based intervention (EBI) selection and implementation process was feasible for use with Phase 2 HCs, one HC in each state piloted the experience, with its matched control HC delaying the implementation process for 12 months. Starting in month 6 of year 1, the Implementation Team in each early pilot HC, with support from the affiliated project team, chose which CRC EBIs to adapt and implement at each level (i.e., patient, provider, HC, community), based on those presented to them to address their respective barriers to CRC screening and follow-up. Project staff worked closely with HC champions to successfully implement the EBIs chosen and track and report outcomes. Providers and support staff were trained in their respective roles by project staff [26–34].

Phase 2 (implementation)

During the implementation phase, project staff repeat the rollout process described previously in both HC and community settings of the remaining eight counties according to the timeline presented in Fig. 1.

Sustainability

Within Phase 2, after the Active Implementation period concludes, the HC and counties enter the Sustainability Period, which ranges from 6 to 24 months depending on when the intervention was introduced (Fig. 1). During this period, project staff allow healthcare provider teams and local community groups to manage the intervention. Staff at the respective universities are available for consultation and troubleshooting, but the goal of this period is to determine how the MLI works in real-world settings. HC champions lead this effort in their HC, and HCs receive a stipend to defray implementation administrative costs. Community champions lead non-healthcare provider-based activities.

Dissemination of the MLI process can be accomplished by other Appalachian-based and/or rural HC and community partners through adapting materials and messages from the current study, and an accompanying project website and toolkit/resource guide may be developed to describe project components, implementation strategies, vignettes, and evaluation methods. In addition, the investigators comply with all Cancer MoonshotSM Resource Sharing guidance, prioritizing dissemination of results in high impact publications and at national conferences and provide technical assistance as needed to other interested partners.

DATA SOURCES, OUTCOME MEASURES AND ANALYSIS PLAN

Table 2 summarizes the outcomes, definitions, data sources, and data collection period, by intervention level.

Table 2

Intervention outcome measures

OutcomeDefinitionData sourcesData collection periodLevel of data analysis
Health Center CRC screeningNumerator: # of patients completed CRC screening
Denominator: # of patients eligible for CRC screening
EHR
UDS measures
AnnuallyPatient
Health centre
County CRC screeningNumerator: # participants self-reported CRC screening
Denominator: # participants completed phone survey
Telephone surveyAnnuallyCommunity
Time to follow-up# of days between positive stool-based test result available to provider and completion of follow-up colonoscopyEHRAnnuallyPatient
Health centre
Clinic CRC follow-upNumerator: # of patients completed follow-up colonoscopy within 6 months of positive stool-based test
Denominator: # of patients with positive stool-based test requiring follow-up colonoscopy
EHRAnnuallyPatient
Health centre
Cost-effectivenessProgram implementation costs (non-research, both labour and non-labour)Cost tracking toolBiannuallyHealth centre
OutcomeDefinitionData sourcesData collection periodLevel of data analysis
Health Center CRC screeningNumerator: # of patients completed CRC screening
Denominator: # of patients eligible for CRC screening
EHR
UDS measures
AnnuallyPatient
Health centre
County CRC screeningNumerator: # participants self-reported CRC screening
Denominator: # participants completed phone survey
Telephone surveyAnnuallyCommunity
Time to follow-up# of days between positive stool-based test result available to provider and completion of follow-up colonoscopyEHRAnnuallyPatient
Health centre
Clinic CRC follow-upNumerator: # of patients completed follow-up colonoscopy within 6 months of positive stool-based test
Denominator: # of patients with positive stool-based test requiring follow-up colonoscopy
EHRAnnuallyPatient
Health centre
Cost-effectivenessProgram implementation costs (non-research, both labour and non-labour)Cost tracking toolBiannuallyHealth centre
Table 2

Intervention outcome measures

OutcomeDefinitionData sourcesData collection periodLevel of data analysis
Health Center CRC screeningNumerator: # of patients completed CRC screening
Denominator: # of patients eligible for CRC screening
EHR
UDS measures
AnnuallyPatient
Health centre
County CRC screeningNumerator: # participants self-reported CRC screening
Denominator: # participants completed phone survey
Telephone surveyAnnuallyCommunity
Time to follow-up# of days between positive stool-based test result available to provider and completion of follow-up colonoscopyEHRAnnuallyPatient
Health centre
Clinic CRC follow-upNumerator: # of patients completed follow-up colonoscopy within 6 months of positive stool-based test
Denominator: # of patients with positive stool-based test requiring follow-up colonoscopy
EHRAnnuallyPatient
Health centre
Cost-effectivenessProgram implementation costs (non-research, both labour and non-labour)Cost tracking toolBiannuallyHealth centre
OutcomeDefinitionData sourcesData collection periodLevel of data analysis
Health Center CRC screeningNumerator: # of patients completed CRC screening
Denominator: # of patients eligible for CRC screening
EHR
UDS measures
AnnuallyPatient
Health centre
County CRC screeningNumerator: # participants self-reported CRC screening
Denominator: # participants completed phone survey
Telephone surveyAnnuallyCommunity
Time to follow-up# of days between positive stool-based test result available to provider and completion of follow-up colonoscopyEHRAnnuallyPatient
Health centre
Clinic CRC follow-upNumerator: # of patients completed follow-up colonoscopy within 6 months of positive stool-based test
Denominator: # of patients with positive stool-based test requiring follow-up colonoscopy
EHRAnnuallyPatient
Health centre
Cost-effectivenessProgram implementation costs (non-research, both labour and non-labour)Cost tracking toolBiannuallyHealth centre

Data sources

Electronic health records

De-identified patient-level data will be requested from HCs annually. Screening rates will be examined by age, gender, race/ethnicity, residence, and payer status. Follow-up and referral-to-care for those with positive screening results and missed opportunities for CRC screening will be assessed at the HC-level.

Telephone survey

All county-level CRC screening rates are being obtained from a county-level CRC screening behavioral assessment like CDC’s BRFSS [5]. Telephone surveys (landline and cell phone) are being conducted in each of the 12 study counties. The survey vendor, targets households in the 12 counties that have at least one resident aged 50–74. All interviewers completed extensive basic interviewer and study-specific training and are certified in Human Subjects Protection.

Men and women aged 50–74 are invited to participate in the telephone interviews. The interviews average 10 min in length as measured by the survey vendor and use standard BRFSS socio-demographic (e.g., sex, age, ethnicity/race, education, income, employment), access to health care, CRC screening (e.g., stool blood testing, colonoscopy and sigmoidoscopy, frequency since last CRC test), and barriers to CRC screening questions.

Provider surveys

Healthcare providers and staff who take part in an education session complete a self-administered questionnaire (pre-test) that is adapted from a previous survey used for university professional educational purposes. The survey focuses on CRC knowledge, beliefs, attitudes, and practices that asks about state and national CRC incidence and mortality rates, knowledge about patient concerns regarding CRC screening, and level of knowledge of patient-level predictors of CRC screening (see Appendix A). Providers and staff also provide socio-demographic information (i.e., age, gender, race, ethnicity, education, and job title). After the session, providers and staff complete a post-test. Notes are used to examine the ­fidelity of the sessions (e.g., the presence of participants during the entire session).

Implementation outcomes

Implementation outcomes from the Proctor Conceptual Model (see Fig. 2) [38] will be assessed by a combination of methods, including provider/staff surveys, observations at the HC and examination of EHR data (all of which are described in greater detail below). To assess feasibility, acceptability, and appropriateness of the ACCSIS initiative in general and each implemented strategy specifically (e.g., patient education, mailed FIT, etc.), HC champions will complete three short validated instruments (i.e., Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure) focused on these outcomes at the end of the active implementation period and at the end of the study period [37]. Fidelity will be examined by periodic observation of healthcare provider and community on-site training and related intervention components and document observations with related notes. To assess penetrance at the provider-level, data on the number of providers using the strategies (based on self-report) will be compared to the number of providers who attend the education session and/or participate in academic detailing sessions. For HC-level penetrance, data on the number of strategies implemented and sustained over time will be assessed. Costs associated with implementation will be tracked by the HC champion weekly for four consecutive weeks, twice each year, in both the active implementation year and the first year of sustainability. Time spent on and itemized costs associated with specific CRC screening and follow-up promotion activities will be documented. The early and delayed HCs will be compared between pairs and the intervention effect will also be examined across all pairs. Adoption will be assessed through routine HC monitoring calls. To assess maintenance (of the intervention), logistic regression models with random HC effects to compare odds of CRC screening at the end of the sustainability period to odds of CRC screening at the end of the implementation period will be employed. Assuming a positive intervention effect, an odds ratio greater than or equal to one will support maintenance of the intervention. Data for these analyses will come from the HCs’ EHRs and thus will be comprised of all patients within the specified age range seen at the HCs over that period. The analysis of maintenance will be stratified by intervention arm with primary interest being in the comparison within the Early Intervention group.

Study outcomes.
Fig 2

Study outcomes.

Service outcomes

Service outcomes will be assessed as appropriate. Efficiency will be determined by assessing the time required to implement the program overall and by component at each HC. While CRC screening has been studied for many years and its safety profile established, safety will be assessed by keeping track of any adverse reactions reported from CRC screening. Equity will be assessed by examining the rates of CRC screening by gender and race, as applicable, as well as age. Timeliness in this project will be assessed in two ways—first, CRC screening adherence after a Cologuard referral or FIT is provided and second, time to follow-up after a positive screen.

Client outcome

The client outcome of knowledge will be assessed through responses on the pre-posttests completed during the provider education session. The other client outcomes of satisfaction and function will not be assessed.

Data analysis

HC-level outcome primary analysis

Data from the participating HC from each of the eight counties (four pairs) that were randomized to active study group will be used; the remaining four counties in Pair 1 will not be included in the primary analysis because they were not randomized, but they will be included in a subsequent sensitivity analysis. The primary outcome will be county-level CRC screening rate, combined across all testing modalities. Following an approach similar to that described by Pennell et al. [39] and implemented in a previous GRT [40], rates will be computed at the HC-level and an ANCOVA model will be used to compare the change in screening rate between the early and delayed intervention counties in each pair. HC will be the unit of analysis for the model: the dependent variable will be HC-level change in CRC screening rate and the ­independent variables will be the study group, baseline HC-level CRC screening rate, and a random pair effect to account for the paired design. Adjustment for the baseline rates will control for any differences by intervention arm due to chance and increases power by decreasing the standard error of the treatment effect [41]. The analysis incorporates weights described by Johnson et al. [42] to account for differences in the number of patients across health systems. To avoid potential biases in intervention effect estimates caused by imbalance in baseline factors, comparisons of county-level measures by arm will be carried out (e.g., incidence, mortality, percent of population screened, median household income, and percent below poverty level). Any factors which differ by a meaningful amount across study arms will be considered potential confounders and included as covariates in the ANCOVA model. Assuming that the assumptions of the ANCOVA are satisfied and no unmeasured baseline characteristics differ between the early and delayed intervention counties, this analytical approach will ensure an unbiased estimate of the intervention effect over the study period.

The sample size justification for this project is based on the primary outcome analysis: an ANCOVA model comparing the counties within each pair with respect to change (pre–post early intervention period) in the HC-level CRC screening rate adjusting for the baseline rate. The calculation utilized a power formula for group-level ANCOVA analyses of data from GRTs [39]. Based on current screening data [43], an average CRC screening rate of 62% is expected across all HCs at baseline and among the delayed intervention HCs following the early intervention period. Thus, assuming an ICC of 0.012 (calculated using current county-level screening rates) [43], a two-sided type-I error rate of 5%, a moderate correlation of 0.5 between the baseline and post-early intervention rates, and no benefit of accounting for pairing in the analysis, a sample size of eight randomized counties and 1300 50- to 74-year-olds/HC at each time point (baseline and post-early intervention) provides 85% power to detect an increase in CRC rate to 74% in the early intervention group. This 12-point percentage increase in CRC screening is based on a previous study in Appalachia [13].

County-level outcome analysis

The secondary outcome for this project will be the change in rate of CRC screening at the county-level from baseline to the end of the 12-month Active Implementation Period for the Early Intervention County in each pair. Like the HC-level outcome, data from eight counties (four pairs) randomized to the active study group will be used; the remaining four counties will be included in a subsequent sensitivity analysis. Screening rates will be determined using data collected by telephone interview. The effectiveness of the intervention will be assessed by comparing the changes in screening rate over the early intervention period in each pair, adjusting for baseline rates.

Cost-effectiveness analysis

Cost-effectiveness analyses will be carried out after the collection of outcome data. The cost-effectiveness analyses will be conducted in three broad steps. A cost identification analysis will be conducted initially. These analyses will be from a payer perspective rather than the broader societal perspective. All costs of each intervention implementation will be considered, including those for patient identification, screening tests, personnel training/implementation, staffing, EHR data collection, and administrative costs. Cost analysis will be limited to intervention costs. Costs will be discounted by the consumer price index to ensure comparable estimates, given the differences in timing of implementation in the HCs and their multi-year follow-up periods. Second, the results of the cost analysis will be combined with the outcome measures to establish the cost per desirable outcome. Specifically, the overall incremental costs associated with the intervention program will be characterized, compared to no program or a delayed program.

Sensitivity analyses will also be performed. Because participants from different age groups will have varied probability of accepting a new program, the analysis will be carried out by groups of participants using 10-year age intervals. In addition, analyses using various effectiveness measures for an intervention (aggregate costs do not change) will be carried out. Monte Carlo-based sensitivity analyses will be conducted to characterize variance in the cost and QALY estimates. Acceptability curves will be plotted for each set of sensitivity analyses. The TreeAge Pro software will be used to generate bootstrap standard errors for cost-effectiveness measures.

DISCUSSION

The goal of the current study is to examine the sustainability of long-term, multilevel interventions in rural Appalachia [44]. Within that context, this study is particularly novel in that interventions are not implemented within a single health care system, but rather through partnerships with 12 small, Appalachian primary care HCs that are connected to different health care systems. HCs were selected because they are (a) located in Appalachian OH and KY counties with disparate CRC incidence and mortality rates and (b) willing/able to participate in research. To that end, this project is targeted specifically to populations at greatest need—rural primary care HCs that often get neglected in the traditional research-to-practice pipeline. Thus, ACCSIS Appalachia is aligned with NCI’s MoonshotSM goal of improving equity in cancer control among disparate populations [45].

The unique approach to implementation science of ACCSIS Appalachia is not, however, without limitations. First, HCs have varying levels of research capacity due, in part, to staffing and budgetary limitations, making sustainment of ­project activities beyond the project timeline a salient concern. To address this barrier, the investigators (i) identified committed and experienced HC champions [46], (ii) engage them throughout all phases of the research process to help build trust [44] and (iii) actively encourage them to select and adapt EBIs that are both implementable and locally relevant within their HC context. Although this approach may yield legitimate concerns about EBI fidelity, previous research in the region shows that community members are highly receptive of this approach and that it can still produce behavioral change [45, 47]. Furthermore, the use of champions has been shown to lead to implementation success in most settings [48]. Although adaptations were not measured in this study because they are not part of the Proctor model, we have been collecting adaptation data using Wiltsey Stirman’s FRAME model to determine the appropriate balance of fidelity versus modification to ensure EBI success that will be reported in an upcoming manuscript [49, 50].

A second challenge is that, due to the structure of the project, standardization of project design is difficult, if not impossible. Since each HC is selecting EBIs based on needs and abilities and further adapting them for local relevance, the project design will likely reflect 12 unique projects. While this approach may result in “messy implementation”, it is reflective of real-world needs, and it centers the HC and communities in developing and implementing their EBIs, making maintenance more feasible. Furthermore, it is uncommon for implementation science projects to include multiple, non-connected health care practices, and as a result project findings hold the potential to be highly significant, particularly to researchers and practitioners working in rural and underserved areas where tailored approaches are both desirable and necessary.

Perhaps the greatest challenge to the unconventional project approach and design is that HC partners have different EHR systems, making outcomes tracking difficult. Research suggests that rural practices often underutilize their EHR capacity compared to urban practices [51] so it is necessary for the project team to provide appropriate technical assistance to ensure that HC can use full EHR functionality to track and measure CRC screening rates and follow-up. Tracking follow-up can be tricky, particularly when patients are sent to specialists in different health systems—a common occurrence in rural Appalachia.

While the approach and focus of the project have great potential to impact CRC screening rates and reduce CRC mortality rates in these areas, the rise of COVID-19 introduced additional unanticipated limitations. Federal and state mandates, policies, and precautions, followed by a focus on testing and then vaccination paused study activities in the HC partners remotely, including implementation planning and delivery of educational sessions. Patients are hesitant to receive anything in paper form. Hospitals and outpatient surgery centers, where many colonoscopy procedures are performed, temporarily halted elective procedures for several months and experienced a significant backlog of patients needing appointments once they resumed pre-pandemic operation. Many HCs also experienced staff furloughs, and some are still not fully staffed. Moreover, the demands of testing, treating, and vaccinating for COVID-19 resulted in CRC screening becoming a lesser priority while HC staff adjusted to accommodate critical priorities. Thus, the project team had to adapt to remote delivery, meetings, and follow-up, and HC staff turned to calling patients to offer FIT testing. Implementation is delayed in most HCs, causing the original timeline to be pushed back by three months.

Lessons learned from this project could be impactful in geographic areas in need of research dissemination and address critical gaps in rural implementation science research, particularly regarding long-term sustainment of multilevel projects in low-resource areas [44]. Small rural HCs are often reticent to participate in research because of legitimate concerns about capacity [44]. This project can contribute to changing that paradigm by showcasing that participation in research (a) does not need to be overly burdensome and (b) can be adaptive to local needs and HC abilities. In addition, the investigators will continue to build and develop rapport with HC and community partners. This is particularly important in Appalachia, where the adoption of appropriate health communication is critical in promoting behavioral change [13].

If effective, the approach used by this project is going to be disseminated to healthcare providers and community partners throughout Appalachia to reduce the significant burden of CRC. Project data (including both clinical and implementation outcomes) will be shared with the NCI data repository and made available to other ACCSIS sites to promote trans-ACCSIS collaboration and identification of cross-cutting barriers and facilitators to implementation.

Acknowledgements

The authors acknowledge the contributions of our collaborating health centers, other ACCSIS centers and the coordinating center, as well as our devoted Healthcare Center and Community Champions of ACCSIS Appalachia: Jeanne Jellison, RN (Ohio Hills Health Services; Quaker City, OH); Michael D. Sarap, MD, FACS (Southeastern Med; Cambridge, OH); Tonia A. Bivens, BSN, RN, CCM (Lewis County Primary Care Center, Inc. dba PrimaryPlus; Vanceburg, KY); Tracy D. McGuire, BS (Lewis County Primary Care Center, Inc. dba PrimaryPlus; Vanceburg, KY); Sarah Miller, CMA (HealthSource of Ohio; Seaman, OH); Debbie Ryan (Creating Health Communities Program; Adams County, OH); Marsha McCormick, MS (OSU Extension—Adams County; West Union, OH); Kayla Bowman, RN (Muskingum Valley Health Centers; Malta, OH); Victoria Gallaher (Southeast, Inc.; St. Clairsville, OH); Michael Kingery, RN (Ironton-Lawrence County Area Community Action Organization Family Medical Centers; Ironton, OH); Douglas Carr, MD (Hopewell Health Centers; Logan, OH); Dana Abney, CPC, CEMC (Mercy Health—Powell County Primary Care; Clay City, KY); Teresa Carroll, CMA (Mercy Health—Powell County Primary Care; Clay City, KY); Morgan Fowler (Mercy Health—Marcum and Wallace Memorial Hospital; Irvine, KY); Ryan Freeman, LNHA, MHA (Three Rivers Medical Center; Louisa, KY); Amanda Auxier, BSN, RN (Three Rivers Family Practice—Riverview; Louisa, KY); Angela Preece, RN (Three Rivers Family Practice—Riverview; Louisa, KY); Todd Branham, MHA (Morgan County ARH; West Liberty, KY); Katherine Carter, MHA (Morgan County ARH; West Liberty, KY); Gloria Riggs, RN (St. Claire Family Medicine—Olive Hill; Olive Hill, KY); and Missy Begley, RN (Juniper Health Breathitt County; Jackson, KY); and UVA/UK survey research centers.

Funding

This study is funded by the National Cancer Institute (4UH3CA233282-02) and supported in part by NCI Cancer Center Support Grants (P30CA177558) and (P30CA016058). Dr. Biederman is supported by NCI T32CA229114.

Compliance with Ethical Standards

Conflicts of Interest: Dr. Paskett reports grants to the institution from Pfizer and Merck Foundation for work unrelated to the current project. Dr. Pennell reports research support from Sanofi-Aventis unrelated to the current project.

Human Rights: All portions of this study were approved by the Institutional Review Board of The Ohio State University (#2018C0166).

Informed Consent: Informed consent was obtained from all individual participants included in the study.

Welfare of Animals: This article does not contain any studies with animals performed by any of the author.

Study Registration: The study was pre-registered at clinicaltrials.gov (NCT04427527). Analytic plan pre-registration: The analysis plan was not formally pre-registered.

Data Availability: De-identified data will be available once a manuscript containing data is published.

Analytic code availability: There is not analytic code associated with this manuscript.

Materials availability: Materials used to conduct the study are not publicly available at this time.

References

1.

Appalachian Community Cancer Network
.
The Cancer Burden in Appalachia
. In progress.

2.

Pollard
K
,
Jacobsen
LA
,
Bureau
PR.
The Appalachian Region: A Data Overview from the 2011–2015 American Community Survey
.
Washington, DC
:
Appalachian Regional Commission
;
2017
.

3.

Siegel
RL
,
Liora
S
,
Anthony
R
,
Ahmedin
J.
Where can colorectal cancer screening interventions have the most impact
?
Cancer Epidemiol Biomarkers Prev.
2015
;
24
(
8
):
1151
1156
.

4.

US Cancer Statistics Working Group
.
United States Cancer Statistics: 1999–2014 incidence and mortality web-based report
;
2017
. Available from: www.cdc.gov/uscs. Accessibility verified May 1, 2018.

5.

Centers for Disease Control and Prevention
.
Behavioral Risk Factor Surveillance System.
2014
; Available from: https://www.cdc.gov/brfss/index.html.

6.

Bibbins-Domingo
K
,
Grossman
DC
,
Curry
SJ
, et al. ;
US Preventive Services Task Force
.
Screening for colorectal cancer: US Preventive Services Task Force Recommendation Statement
.
JAMA.
2016
;
315
(
23
):
2564
2575
.

7.

Marshall
JL
, et al. .
Health Disparities in Appalachia
.
Raleigh, N.C
.:
PDA, Inc., The Cecil G. Sheps Center for Health Services Research, Appalachian Regional Commission
;
2017
.

8.

Bachman
AS
,
Cohen
EL
,
Collins
T
,
Hatcher
J
,
Crosby
R
,
Vanderpool
RC
.
Identifying communication barriers to colorectal cancer screening adherence among Appalachian Kentuckians
.
Health Commun.
2017
;
33
(
10
):
1284
1292
doi: 10.1080/10410236.2017

9.

Hatcher
J
,
Dignan
MB
,
Schoenberg
N.
How do rural health care providers and patients view barriers to colorectal cancer screening? Insights from Appalachian Kentucky
.
Nurs Clin N Am.
2011
;
46
(
2
):
181
92, vi
.

10.

Royse
D
,
Dignan
M.
Fatalism and cancer screening in Appalachian Kentucky
.
Fam Community Health.
2011
;
34
(
2
):
126
133
.

11.

Royse
D
,
Dignan
M.
Appalachian knowledge of cancer and screening intentions
.
J Cancer Educ.
2009
;
24
(
4
):
357
362
.

12.

Rosenwasser
LA
,
McCall-Hosenfeld
JS
,
Weisman
CS
,
Hillemeier
MM
,
Perry
AN
,
Chuang
CH.
Barriers to colorectal cancer screening among women in rural central Pennsylvania: primary care physicians’ perspective
.
Rural Remote Health.
2013
;
13
(
4
):
2504
.

13.

Dignan
M
,
Shelton
B
,
Slone
SA
, et al. .
Effectiveness of a primary care practice intervention for increasing colorectal cancer screening in Appalachian Kentucky
.
Prev Med.
2014
;
58
:
70
74
.

14.

Portnoy
B.
Healthy People in Rural Areas by the Year 2000, in Sowing Seeds in the Mountains: Community-based Coalitions for Cancer Prevention
. In:
Couto
R
,
Simpson
NK
,
Harris
G
, eds.
National Cancer Institute
:
Bethesda
;
1994
:
102
119
.

15.

Paskett
E
,
Thompson
B
,
Ammerman
AS
,
Ortega
AN
,
Marsteller
J
,
Richardson
DJ.
Multilevel interventions to address health disparities show promise in improving population health
.
Health Aff (Millwood).
2016
;
35
(
8
):
1429
1434
.

16.

Warnecke
RB
,
Oh
A
,
Breen
N
, et al. .
Approaching health disparities from a population perspective: the National Institutes of Health Centers for Population Health and Health Disparities
.
Am J Public Health.
2008
;
98
(
9
):
1608
1615
.

17.

Tessaro
I
,
Mangone
C
,
Parkar
I
,
Pawar
V.
Knowledge, barriers, and predictors of colorectal cancer screening in an Appalachian church population
.
Prev Chronic Dis.
2006
;
3
(
4
):
A123
.

18.

Kelly
KM
,
Phillips
CM
,
Jenkins
C
, et al. .
Physician and staff perceptions of barriers to colorectal cancer screening in Appalachian Kentucky
.
Cancer Control.
2007
;
14
(
2
):
167
175
.

19.

Schoenberg
NE
,
Eddens
K
,
Jonas
A
, et al. .
Colorectal cancer prevention: perspectives of key players from social networks in a low-income rural US region
.
Int J Qual Stud Health Well-being.
2016
;
11
:
30396
.

20.

Knight
JR
,
Kanotra
S
,
Siameh
S
,
Jones
J
,
Thompson
B
,
Thomas-Cox
S.
Understanding barriers to colorectal cancer screening in Kentucky
.
Prev Chronic Dis.
2015
;
12
:
E95
.

21.

Kruse-Diehr
AJ
,
Oliveri
JM
,
Vanderpool
RC
, et al. .
Development of a multilevel intervention to increase colorectal cancer screening in Appalachia
.
Implement Sci Commun.
2021
;
2
(
1
):
51
.

22.

Donner
A
,
Klar
N.
Design and Analysis of Cluster Randomization Trials in Health Research
.
London
:
Arnold
;
2000
.

23.

Hayes
RJ
,
Moulton
LH.
Cluster Randomized Trials
.
Boca Raton, FL
:
CRC Press
;
2009
.

24.

Murray
D.
Design and Analysis of Group-Randomized Trials
.
New York
:
Oxford
;
1998
.

25.

National Cancer Institute
.
Evidence-based Cancer Control Programs (EBCCP).
Available from: https://ebccp.cancercontrol.cancer.gov/index.do. Accessibility verified May 1, 2018.

26.

Walsh
JM
,
Salazar
R
,
Nguyen
TT
, et al. .
Healthy colon, healthy life: a novel colorectal cancer screening intervention
.
Am J Prev Med.
2010
;
39
(
1
):
1
14
.

27.

Myers
RE
,
Turner
B
,
Weinberg
D
, et al. .
Impact of a physician-oriented intervention on follow-up in colorectal cancer screening
.
Prev Med.
2004
;
38
(
4
):
375
381
.

28.

Lane
DS
,
Messina
CR
,
Cavanagh
MF
,
Chen
JJ.
A provider intervention to improve colorectal cancer screening in county health centers
.
Med Care.
2008
;
46
(
9 Suppl 1
):
S109
S116
.

29.

Curry
WJ
,
Lengerich
EJ
,
Kluhsman
BC
, et al. .
Academic detailing to increase colorectal cancer screening by primary care practices in Appalachian Pennsylvania
.
BMC Health Serv Res.
2011
;
11
:
112
.

30.

Wang
A
,
Rachocki
C
,
Shapiro
JA
,
Issaka
RB
,
Somsouk
M.
Low literacy level instructions and reminder calls improve patient handling of fecal immunochemical test samples
.
Clin Gastroenterol Hepatol.
2019
;
17
(
9
):
1822
1828
.

31.

Division of Cancer Prevention and Control Centers for Disease Control and Prevention
.
Screen for Life: National Colorectal Cancer Action Campaign 2020
.
Washington, DC
:
US Department of Health and Human Services
;
2020
.

32.

Castaneda
SF
,
Bharti
B
,
Rojas
M
, et al. .
Outreach and inreach strategies for colorectal cancer screening among Latinos at a federally qualified health center: a randomized controlled trial, 2015–2018
.
Am J Public Health.
2020
;
110
(
4
):
587
594
.

33.

Redwood
D
,
Provost
E
,
Asay
E
,
Ferguson
J
,
Muller
J.
Giant inflatable colon and community knowledge, intention, and social support for colorectal cancer screening
.
Prev Chronic Dis.
2013
;
10
:
E40
.

34.

Green
BB
,
Wang
C-Y
,
Anderson
ML
, et al. .
An automated intervention with stepped increases in support to increase uptake of colorectal cancer screening: a randomized trial
.
Ann Intern Med.
2013
;
158
(
5 Pt 1
):
301
311
.

35.

Radhakrishnan
K
,
Slone
S
,
Shelton
B
,
Jarrett
T
,
Pearce
K
,
Dignan
M
.
Persistent poverty reduces effectiveness of academic detailing in increasing screening colonoscopy in Appalachian Kentucky
.
Arch Epid Pub Health
2020
;
1
:
1
-
7
.

36.

Vanderpool
RC
,
Moore
SC
,
Stradtman
LR
,
Carman
AL
,
Kurgat
HL
,
Fain
P.
Adaptation of an evidence-based intervention to improve preventive care practices in a federally qualified health center in Appalachian Kentucky
.
J Health Care Poor Underserved.
2016
;
27
(
4A
):
46
52
.

37.

Weiner
BJ
,
Lewis
CC
,
Stanick
C
, et al. .
Psychometric assessment of three newly developed implementation outcome measures
.
Implement Sci.
2017
;
12
(
1
):
108
.

38.

Proctor
EK
,
Landsverk
J
,
Aarons
G
,
Chambers
D
,
Glisson
C
,
Mittman
B.
Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges
.
Adm Policy Ment Health.
2009
;
36
(
1
):
24
34
.

39.

Pennell
ML
,
Hade
EM
,
Murray
DM
,
Rhoda
DA.
Cutoff designs for community-based intervention studies
.
Stat Med.
2011
;
30
(
15
):
1865
1882
.

40.

Krok-Schoen
JL
,
Katz
ML
,
Oliveri
JM
, et al. .
A media and clinic intervention to increase colorectal cancer screening in Ohio Appalachia
.
Biomed Res Int.
2015
;
2015
:
943152
.

41.

Murray
DM
,
Lee Van Horn
M
,
Hawkins
JD
,
Arthur
MW.
Analysis strategies for a community trial to reduce adolescent ATOD use: a comparison of random coefficient and ANOVA/ANCOVA models
.
Contemp Clin Trials.
2006
;
27
(
2
):
188
206
.

42.

Johnson
JL
,
Kreidler
SM
,
Catellier
DJ
,
Murray
DM
,
Muller
KE
,
Glueck
DH.
Recommendations for choosing an analysis method that controls Type I error for unbalanced cluster sample designs with Gaussian outcomes
.
Stat Med.
2015
;
34
(
27
):
3531
3545
.

43.

National Cancer Institute
.
State Cancer Profiles
. https://statecancerprofiles.cancer.gov/. Accessibility verified January 2, 2021.

44.

Weaver
SJ
,
Blake
KD
,
Vanderpool
RC
,
Gardner
B
,
Croyle
RT
,
Srinivasan
S.
Advancing rural cancer control research: National Cancer Institute efforts to identify gaps and opportunities
.
Cancer Epidemiol Biomarkers Prev.
2020
;
29
(
8
):
1515
1518
.

45.

Oh
A
,
Vinson
CA
,
Chambers
DA.
Future directions for implementation science at the National Cancer Institute: Implementation Science Centers in Cancer Control
.
Transl Behav Med.
2021
;
11
(
2
):
669
675
.

46.

Shea
CM.
A conceptual model to guide research on the activities and effects of innovation champions
.
Implement Res Pract.
2021
;
2
.

47.

Vanderpool
RC
,
Gainor
SJ
,
Conn
ME
,
Spencer
C
,
Allen
AR
,
Kennedy
S.
Adapting and implementing evidence-based cancer education interventions in rural Appalachia: real world experiences and challenges
.
Rural Remote Health.
2011
;
11
(
4
):
1807
.

48.

Miech
EJ
,
Rattray
NA
,
Flanagan
ME
,
Damschroder
L
,
Schmid
AA
,
Damush
TM.
Inside help: an integrative review of champions in healthcare-related implementation
.
SAGE Open Med.
2018
;
6
:
2050312118773261
.

49.

Carvalho
ML
,
Honeycutt
S
,
Escoffery
C
,
Glanz
K
,
Sabbs
D
,
Kegler
MC.
Balancing fidelity and adaptation: implementing evidence-based chronic disease prevention programs
.
J Public Health Manag Pract.
2013
;
19
(
4
):
348
356
.

50.

Wiltsey Stirman
S
,
Baumann
AA
,
Miller
CJ.
The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions
.
Implement Sci.
2019
;
14
(
1
):
58
.

51.

Rumball-Smith
J
,
Fromkin
J
,
Rosenthal
B
,
Shane
D
,
Skrbin
J
,
Bimber
T
,
Berger
RP
.
Implementation of routine electronic health record-based child abuse screening in general emergency departments
.
Child Abuse Negl.
2018
;
85
:
58
67
.

Author notes

Dr. Robin Vanderpool was formerly with the University of Kentucky College of Public Health and co-investigator on the ACCSIS Appalachia grant during Year 1; she is now employed by the National Cancer Institute.

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://dbpia.nl.go.kr/pages/standard-publication-reuse-rights)