Abstract
The benefits of new clinical research developments often take years to reach patients. As such, the Departments of Defense (DoD) and Veterans Affairs built the Practice-Based Implementation (PBI) Network as an infrastructure to facilitate more rapid translation of psychological health (PH) research into clinical practice changes to improve the quality of care for military and Veteran patients. To regularly identify research findings appropriate for enterprise implementation, the DoD PBI Network developed a model aligned with the Consolidated Framework for Implementation Research to select and pilot PH practice change. Within this model, practice change pilots were selected following a survey of field clinicians, a public call for proposals, annual meeting of implementation science subject matter experts, and final pilot selection by PH strategic leaders. These components facilitated commitment and engagement from Military Health System PH leadership, as well as clinicians, leading to increased stakeholder buy-in and efficiency with selecting and piloting PH practice change. The DoD PBI Network model has been refined for future PH research translation pilots. It serves as a first operational model for annual implementation of PH research findings in the DoD and may be of use to other entities engaged in practice change implementation.
INTRODUCTION
Multiple studies reference an average of 17 years for health research findings to reach clinical practice.1,2 In the Department of Defense (DoD), this delay is particularly manifested through challenges in timely dissemination and implementation of evidenced-based guidelines and practices. For instance, while the Department of Veterans Affairs (VA)/DoD Clinical Practice Guideline for the Management of Post-Traumatic Stress (Version 2.0) detailing evidence-based treatments for posttraumatic stress disorder (PTSD) was released in 2010, a 2014 study by the National Academy of Medicine (formerly the Institute of Medicine) found that psychological health (PH) care providers in both departments did not consistently offer these evidence-based treatments to their patient despite available guidelines and related policies.3,4 Similarly, though research has substantiated the benefits of routine outcome monitoring with patients and this practice was recommended in the clinical practice guideline, the same 2014 study found that both DoD and VA did not regularly use outcome measures for patients with PTSD.4,5 There are many factors that can affect the implementation of clinical practices, such as those at the organizational level with regards to leadership support; at the clinic level in terms of available resources and patient care demands; and at the provider level to include time constraints, available related training and consultation, and willingness to alter practice behaviors.6
To address these factors, the DoD and the VA established the Practice-Based Implementation (PBI) Network as an infrastructure to more rapidly translate PH research into clinical practice and in turn facilitate improved quality of care for their patients. Each year, the PBI Network selects an appropriate evidence-based practice change to pilot within the DoD enterprise. These pilot projects allow testing of evidenced-based research in military treatment facilities (MTF) and provide the opportunity to determine facilitators and barriers to PH practice changes that inform recommendations for enterprise-wide adoption. This paper describes the practice change pilot selection process within the overall sustainment model for the DoD PBI Network.
History of the DoD PBI Network
In 2010, the DoD/VA Integrated Mental Health Strategy Strategic Action #26 Task Group formed to examine the translation of PH research findings into routine clinical practice. Following review of processes, systems, and tools by which research findings were translated into practice, the Task Group identified the absence of a structure to improve coordination and promote more collaborative processes with an intended end state of more rapid translation. To address this need, a pilot was developed to establish the DoD/VA PBI Network in mental health, funded through a two-year DoD/VA Joint Incentive Fund (JIF). Through the PBI Network, tools and practice changes developed from research findings could be implemented via a pilot at the clinic level, assessed for barriers and facilitators to practice change, and then adjusted based on lessons learned prior to broader dissemination across the enterprise.
The PBI Network consists of clinicians at selected sites throughout DoD and VA, clinic leaders and site champions (i.e., early adopters who serve as local sources of individual knowledge and confidence about a practice change) a web-based platform for communication and knowledge exchange, and PBI Network subject matter expert teams located at the DoD Psychological Health Center of Excellence (PHCoE, previously under the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury) and the VA National Center for Posttraumatic Stress Disorder (PTSD). The PBI Network subject matter experts develop and deliver training and facilitation to support practice change at the different sites.
To evaluate the effectiveness of the DoD/VA PBI Network, the development team sought to address the evidence-based practice of routine outcome monitoring. The DoD/VA PBI Network brought together clinicians and clinic managers at 32 mental health clinics across VA and DoD to examine resulting practice changes through the use of measurement-based care for PTSD. Specifically, an implementation pilot was conducted to promote use of the PTSD Checklist (PCL), a self-report measure that assesses the presence and severity of symptoms of PTSD and can be used to monitor symptom changes in patients during and after treatment.3 Through the pilot, the DoD/VA PBI Network and the field-based clinics and clinicians identified system-specific barriers to adoption of this outcome monitoring and solutions to promote more widespread use of the evidence-based practice in DoD and VA clinics. Given the positive benefits observed through the DoD/VA PBI Network (e.g., accelerated and sustained practice change through increased use of the PCL to monitor PTSD treatment), sustainment was recommended at the conclusion of the JIF funded project.7 In 2016, DoD PH strategic leaders endorsed this sustainment and committed to annual selection of a practice change pilot to implement via a DoD PBI Network.
Development of the PBI Network Model
The initial PBI Network model was based on the Promoting Action on Research Implementation in Health Services8 and Quality Enhancement Research Initiative9 research translation models. However, to develop a sustainment model, the DoD PBI Network also used the Consolidated Framework for Implementation Research (CFIR)5 for foundational guidance. The CFIR was developed by synthesizing definitions and constructs from 19 implementation theories, resulting in a meta-theoretical framework thought to represent a professional consensus on implementation science concepts. It provides structure for the identification of potential influences on implementation and a framework for synthesizing and building knowledge about what works where, across multiple settings. CFIR comprises five major domains: the intervention, inner setting, outer setting, the individuals involved in the implementation process, and the process by which implementation is accomplished.5 These domains interact to influence implementation effectiveness. Each domain consists of multiple constructs. To positively leverage the CFIR, the DoD PBI Network carefully considered constructs from each of the domains, as well as an overarching emphasis on collaboration and active involvement of key stakeholders (e.g., clinicians, researchers, and behavioral health leaders at multiple levels of the organization) in model development. CFIR constructs from the intervention and inner and outer setting domains primarily applicable to the practice change pilot selection part of the sustainment model are discussed in the remainder of this section. Fewer details are presented on the other two CFIR domains – characteristics of the individuals and process – in light of their greater emphasis during the implementation phase of the practice change pilot.
Intervention
The intervention source can influence success of implementation of an intervention. Interventions may arise from sources internal or external to an organization.10 Interventions arising from internal sources (e.g., a provider from within a clinic or a site champion of the DoD PBI Network team) tend to spontaneously produce adaptations that spread through an organization organically. For instance, regular use of a substance misuse screening measure by one mental health provider within a clinic experiencing an increase in number of patients with co-occurring substance use disorders may be a naturally adopted practice by others within the clinic as it addresses a local problem. Externally developed interventions (e.g., at the Service- or DoD-level external to the local clinics and the DoD PBI Network) tend to be disseminated through centralized and controlled processes. While leadership support is essential to successful implementation, especially in a hierarchical organization such as the DoD, overly authoritative dissemination methods may reduce the chance a practice change is successfully implemented.11 The DoD PBI Network sought to balance the potentially negative impacts of authoritative external intervention sources through dissemination of an open call for proposals to begin the process of practice change pilot selection. This open call would allow for solicitation of both external and internal intervention sources to be considered. The final selection of the practice change pilot would be conducted by DoD senior leaders external to the clinics and the DoD PBI Network. However, in 2013 the DoD PBI Network developed a tool, “The Provider Needs and Assessment Survey,” to collect information regarding the PH needs of the DoD from providers so that additional internal source input was available for consideration in the selection of the practice change for further balancing effects.
Another pertinent construct within the intervention domain is the evidence strength and quality, as strong evidence is a necessary but insufficient component of clinical practice change.12 While much of the focus in the evidence-based practice movement has been on quality of the scientific evidence, clinicians consider multiple sources of evidence beyond research and published guidelines, including clinical experiences, patient experiences, needs and preferences, local practice information, and attributes of the proposed practice change.8 In order to allow these multiple sources of evidence to inform implementation pilot selection, the DoD PBI Network opted to convene an annual work group of subject matter experts to review implementation pilot proposals and rank them with respect to scientific merit and feasibility for implementation.
Inner Setting
In the CFIR, attention is paid to multiple aspects of the inner setting, including implementation climate and readiness for implementation. Implementation climate is a function of collective receptivity of individuals to implementation as well as the organizational rewards, support, and expectations around implementation. It involves sub-constructs of tension for change, compatibility, relative priority, organizational incentives and rewards, and goals and feedback.10
As noted, the DoD PBI Network tool, “The Provider Needs and Assessment Survey,” was designed to collect information from providers and provide critical data about implementation climate at the clinic level. Data from this survey reflect provider perceptions related to the implementation climate sub-constructs of relative priority and compatibility. Relative priority represents a collective assessment of the importance of an intervention. Compatibility refers to individuals’ perceptions of how the intervention fits within their values and within the system’s existing workflow.10 Through the survey, the DoD PBI Network is better able to understand priorities of Military Health System (MHS) providers and potential challenges to practice change implementation, can gain valuable feedback from the field regarding perceived needs of providers, and can begin to help shape priorities for PH research translation.
Readiness for implementation is comprised of tangible and immediate indicators of organizational commitment to a decision to implement an intervention, to include leadership engagement, available resources, and access to information and knowledge.10 In the practice change pilot selection and overall sustainment model, the DoD PBI Network focused considerable attention on the need for early leadership engagement. Requesting that DoD leadership make the final selection of an intervention for a practice change pilot creates the opportunity to make buy-in more likely. The selected intervention would be presumed in alignment with perceived needs at the leadership level, and subsequently facilitate implementation commitment at local levels within the larger organization.
Outer Setting
The outer setting comprises the economic, political, and social context within which an organization resides.10 Practice change factors on the periphery such as patient needs and resources and external policy and incentives were of particular relevance in the development of the DoD PBI Network sustainment model. Patient needs and resources are the extent to which patient needs, as well as barriers and facilitators to meet those needs, are accurately known and prioritized by the organization.13 Knowledge of patient needs and resources is essential as patient-centered organizations are more likely to implement change effectively.14 “The Provider Needs and Assessment Survey” garners this information as responses reflect perceptions of needs and resources for patients as well as for providers. External policy and incentives are external strategies to spread interventions, such as changes in policy and regulations (governmental or other central entity), external mandates, recommendations and guidelines, pay-for-performance, collaboratives, and public or benchmark reporting.13 In the DoD, change often occurs following implementation of a policy directive or instruction. As such, the DoD PBI Network considered the importance of codifying the sustainment model to promote practice change through development of a related memorandum of understanding, or policy, and expects to disseminate the policy within 2018. This policy will document the DoD PBI Network model and include processes for collaboration, identification, and implementation of PH practice change pilots in MTFs to improve clinical programs, policies, and practices.
Characteristics of the Individual
Individual characteristics such as knowledge and beliefs about the intervention and level of self-efficacy can facilitate or hinder practice chance.8 Knowledge and beliefs about the intervention include attitudes toward and value placed on the intervention, and familiarity with facts, truths, and principles related to the intervention. To address this factor, one of the first steps within the DoD PBI Network sustainment model involves the identification of site champions early in the practice change pilot. Site champions are often motivated early adopters who serve as local sources of individual knowledge and confidence about the practice change, thus improving buy-in across their clinic or site through their influence. Another pertinent individual characteristic is the level of self-efficacy that an individual may have with a potential practice change. Individuals with high self-efficacy are more likely to embrace a practice change and commit to it even when faced with obstacles.8 The DoD PBI Network developed a web-based platform for communication and knowledge exchange to increase self-efficacy, as well as promote ongoing collaboration, among pilot site champions, clinic leaders, and other relevant individuals. Facilitation calls during the implementation phase support self-efficacy of providers through consultation with experts and peers.
Process
The CFIR describes four essential activities of the implementation process common across organizational change models: planning, engaging, executing, and reflecting and evaluating.8 The DoD PBI Network sustainment model addresses these activities primarily during the practice change pilot implementation. Planning, or “the degree to which a scheme or method of behavior and tasks for implementing an intervention are developed in advance, and the quality of those schemes or methods,” is a significant component.13 The DoD PBI Network developed a detailed standardized practice change pilot methodology with continual refinement of extant processes via a feedback loop within the sustainment model. Engaging, executing, and reflecting and evaluating activities occur throughout the practice change pilot implementation. As noted, the DoD PBI Network attempts to engage site champions and clinic leaders through training, regular stakeholder calls, and use of the collaborative web-based platform. Ongoing facilitation teleconferences throughout practice change pilot implementation between the DoD PBI Network team and site champions and clinicians provide the opportunity for reflecting and evaluating, to include qualitative feedback on the process and quality of implementation.
METHODS
A key part of the DoD PBI Network model involves procedures to facilitate annual selection of scientifically sound and clinically relevant PH practice changes to pilot within the DoD (Fig. 1). The practice change pilot selection process is detailed in this section.
DoD PBI Network Model for Implementing Psychological Health Practice Change (Version 1).
Collection of PH Practice Change Pilot Proposals
A formal call for proposals was crafted and disseminated to collect practice change pilot proposals to consider for translation. The DoD PBI Network sought proposals of evidence-based interventions, programs, or practices to support the PH of Service members, Veterans, and/or their families that could be implemented and sustained within the MHS, as well as piloted within an approximate 6-month timeframe. The call for proposals was disseminated via direct email to contacts from the DoD and VA who had previously collaborated with PHCoE. The window for submission of proposals was approximately 6 weeks. Proposal instructions required descriptions of: the background and literature base, to include a problem statement and an objective for the effort; the problem and the proposed solution, including the materials that may need to be developed and implemented to solve the problem; the stakeholder need for the pilot; how the pilot helped to meet the Defense Health Agency (DHA)/PHCoE mission; what the pilot would actually look like and how it could be implemented; and appropriate supporting citations and references.
Initial Examination of PH Practice Change Pilot Proposals
Following closure of the formal proposal submission window, submitted proposals were examined to ensure completion and compliance with instructions. Subsequently, the proposals and their relevant supporting body of evidence were forwarded for review, scoring, and ranking by the designated PH and implementation science experts.
Review, Scoring, and Ranking of PH Practice Change Pilot Proposals
The DoD PBI Network employed the Psychological Health Practice Change Prioritization Work Group (PCPWG) to review, score, and rank the submitted proposals. This work group is comprised of PH and implementation science experts from DoD and external agencies, including the Army Medical Research and Materiel Command, the Uniformed Services University of the Health Sciences, the Center for Deployment Psychology, the National Intrepid Center of Excellence, the National Institutes of Health (NIH), the Substance Abuse and Mental Health Services Administration (SAMHSA), the Walter Reed Army Institute of Research, the VA, and PHCoE. Each member individually evaluated each submitted proposal and then at the annual meeting of the PCPWG, group members discussed the reviewed proposals. Proposals were then re-evaluated, scored, and ranked collectively by the PCPWG upon the following weighted criteria:
Established Need (15%): The need for the proposed solution is established by pre-identified requirements (e.g., VA/DoD clinical practice guidelines, DoD Instructions, etc.) generated by various authorities (e.g., DoD, PHCoE, National Academy of Medicine, and SAMHSA).
Mission Alignment (30%): The solution has the capacity to improve the lives of Service members, Veterans and their families by advancing excellence in psychological health and traumatic brain injury prevention and care.
Quality of Evidence (20%): Evidence exists to support the proposal from at least one well designed, adequately powered, randomized controlled trial or quasi-experimental study that has been published in a peer-reviewed journal.
Solution Maturity (10%): The product is in its near final form and has been tested in at least one population that may or may not represent the targeted demographic.
Feasibility of Implementation (25%): The resources required for implementation are reasonable and likely available.
Proposal scoring was modeled from parts of the NIH proposal scoring system.15 Reviewers scored each proposal on the five criteria using a Likert scale ranging from one (i.e., exceptional) to nine (i.e., poor). Criterion scores were multiplied by that criterion’s assigned weight to produce an impact score for each of the criteria. Impact scores were summed to produce the proposal’s individual overall impact score. Individual overall impact scores were averaged across reviewers to provide a final score. An example scoring form can be found in Appendix 1.
Selection of a PH Practice Change Pilot
Once proposals were scored, they were ranked numerically and presented to a higher level work group within the DoD for pilot selection. Specifically, the DoD PBI Network presented the proposals to the Behavioral Health Clinical Community (BHCC; formerly the DoD Mental Health Work Group), a group comprised of PH leadership from DHA and each of the Services charged with accomplishing goals in support of the MHS Quadruple Aim: improving mental health across the beneficiary population; ensuring that the mental health system of care is accessible, safe, and effective; standardizing processes and sharing resources to lower cost; and increasing the mental health readiness of the force. This group reviewed the rank-ordered list of pilots, reviewed the results of the most recent provider needs survey (2013) to consider the expressed translation needs from the field, and selected the proposal best aligned with MHS PH strategic goals for implementation by the DoD PBI Network in the following calendar year.
RESULTS
For the 2017 implementation pilot, three proposals were received from DoD and VA entities in response to the call, representing different areas of focus within PH. These proposals included: Expanding and Enhancing Mental Health Care: Using Technology to Access, Monitor and Improve Mental Well-Being (“Tech into Care”); Centrally-assisted Telecare for PTSD and Depression; and a Brief Cognitive Behavioral Therapy (BCBT) Replication Trial. The “Tech into Care” proposal aimed to facilitate the use and integration of mobile and web applications to extend and enhance behavioral health interventions offered by DoD and VA providers as a means to improve both patient access and psychological health. The Centrally-assisted Telecare for PTSD and Depression Proposal sought to build upon existing integrated care infrastructure by: (1) centralizing implementation coordination, (2) enhancing care management, (3) adding stepped psychotherapeutic options, and (4) using clinical registries to guide treatment to improve integration of PH care in the MHS. The BCBT Replication Trial aimed to replicate the effectiveness of BCBT for the prevention of suicide attempts in a clinical sample of Marine personnel and identify potential mechanisms of change underlying the effects of BCBT.
The three proposals were forwarded to the PCPWG members for review. The PCPWG members convened to discuss and score the proposals. The BHCC was subsequently provided the practice change pilot proposals and the ranked scores. After deliberation for just over 60 days, the BHCC ultimately selected the “Tech into Care” proposal for the 2017 practice change implementation pilot. Final decision regarding PH practice change pilot selection occurred within 120 days of the closing date of the formal call for proposals. Following selection of the “Tech into Care” proposal, implementation, facilitation, and training plans were developed and the practice change pilot launched within several months at multiple sites across DoD via the PBI Network sites and site champions.
DISCUSSION
Research highlights a time lag of nearly two decades from discovery of health research findings until translation into routine clinical practice.2 As such, it may take many years for evidence-based treatments to reach their intended patients. Factors at the organizational, clinic, and provider level may all impact implementation of clinical practice changes. To address this knowledge to practice gap, the DoD and VA established the PBI Network as a collaborative model to facilitate research translation. The PBI Network initially launched as a joint initiative, though it is sustained separately by each Department. The DoD PBI Network seeks to execute practice change pilots to ensure the latest PH research and evidence-based guidance are rapidly made available to MHS providers through translation into tangible clinical tools and practices. To support annual practice change implementation pilots, the DoD PBI Network developed a sustainment model, which includes a process for practice change pilot selection for which methodology was discussed in this paper. Through development of a model in alignment with the CFIR, the practice change pilot selection process contributed to the success of the subsequent pilot.
Following development, the DoD PBI Network initiated the first application of the sustainment model through the practice change pilot selection. This process consisted of executing a public call for proposals, bringing together subject matter experts via a working group to review and score proposals, and facilitating DoD PH leadership selection of a practice change pilot – the “Tech into Care” pilot – for implementation in 2017. The practice change pilot selection process was completed in a timely manner, within 120 days following close of the call for proposals. The current “Tech into Care” pilot is ongoing and the roll out has thus far been rapid, efficient, and promising. As noted, engagement with leadership and other entities throughout the enterprise in the practice change selection facilitated buy-in to the intervention. The DoD PBI Network has been able to implement this practice change pilot while concurrently beginning the next cycle of practice change proposal selection through use of the sustainment model. This model appears to be the first of its kind for automating regular practice change interventions within MHS governance.
Strengths
Implementation of the sustainment model within the DoD PBI Network contributed to a swift implementation of the practice change pilot chosen for 2017. As DoD leaders were involved prior to implementation, necessary relationships were initiated early on and essential leadership buy-in was achieved from the start. Providing scientifically-vetted proposals from which to select a best fit with enterprise needs garnered credibility for the practice change from senior DoD PH leaders, and enhanced overall improved readiness for implementation in the field.
Limitations and System Challenges
While multiple components within the DoD PBI Network sustainment model (i.e., call for proposals, selection process, PCPWG) were successfully executed, the revised Provider Needs and Assessment Survey has not yet been disseminated due to lengthy administrative delays in obtaining necessary survey licensure approvals.
Future Directions
To further refine the sustainment model, the DoD PBI Network plans to deploy an expanded call for proposals in subsequent years, to include posting the call for proposals to the PHCoE website and promoting the call at conferences, professional networking events, and PHCoE outreach efforts. The team will also lengthen the time window for submission of proposals to permit increased opportunity for entities to submit. This way, DoD implementation science subject matter experts and leadership can be aware of even more potential practice changes to make the most informed practice change pilot selection.
The Provider Needs and Assessment Survey will inform pilot selection through increased understanding of PH needs of the DoD from the viewpoint of the clinician. The DoD PBI Network plans to administer this survey, once approved, to approximately 200 clinical providers. In the interim, the DoD PBI Network has solicited informal provider feedback via ongoing facilitation calls to help inform the 2018 practice change pilot selection process, and leveraged other DoD funded research data to obtain provider feedback from the field. Results of the survey will help to delineate challenges in providing integrated and coordinated PH care across MTFs, identify challenges in implementing potential strategies or systems to improve PH care, and assess barriers to and facilitators for initiating practice changes at MTFs, including organizational and provider factors. Survey data will also provide an opportunity to obtain clinician-perceived needs for practice change and research. Further, those involved in practice change pilot selection will be able to ascertain which proposal(s) most directly align with clinician-identified needs. A practice change that meets a well-established clinical need can leverage aspects of the implementation climate, such as tension to change, compatibility, and relative priority, and would likely be more rapidly adopted.10
The DHA-Procedural Instruction codifying the DoD PBI Network model in policy is still in preparation. This document establishes major stakeholder responsibilities and annual procedures for practice change pilot selection and implementation. Establishing the DoD PBI Network in policy signals the organization’s ongoing commitment to improving care through implementation initiatives. More importantly, it secures the mechanism for the annual execution of practice change pilots, which will enhance the speed at which practice changes may be tested and adopted within the MHS.
CONCLUSIONS
Use of the DoD PBI Network sustainment model as an institutional process reoccurring on an annual basis may serve to both increase the speed and automaticity of piloting PH practice changes, as well as the stakeholder buy-in during practice change implementation. More importantly, it may shorten the time for evidence-based PH treatments to reach MHS patients and improve quality of care. The DoD PBI Network model has been refined for future DoD PH research translation pilots (Fig. 2) and may be of use to other entities engaged in dissemination and implementation within the DoD. This model, with underpinnings in the CFIR and principles of collaboration and active involvement, may promote more rapid translation of research findings into the clinics as well as sustained practice change.
DoD PBI Network Model for Implementing Psychological Health Practice Change (Version 2).
Presentations
Presented as a poster at the 2017 Military Health System Research Symposium, Kissimmee, FL 28–29 August 2017; MHSRS 17–0346.
Funding
Formation of the Practice-Based Implementation Network was originally funded through the Department of Defense (DOD)/Department of Veterans Affairs (VA) Joint Incentive Fund (JIF) for Integrated Mental Health Strategy (IMHS) Strategic Action #26, Practice-Based Implementation Network in Mental Health. Funds were awarded in 2013. This supplement was sponsored by the Office of the Secretary of Defense for Health Affairs.
Acknowledgments
We thank Drs Jorielle Houston and Garnette Cotton for their assistance with editing during manuscript preparation.
Appendix 1. Sample Proposal Scoring Form
PBI Network Proposal Prioritization System and Procedures
The Practice Based Implementation (PBI) Network proposal scoring system will be used by the Psychological Health Practice Change Prioritization Work Group to assess submitted implementation project proposals. Prior to the review meeting, participants should score each proposal using the provided scoring sheet and be prepared to discuss and provide justification at the Work Group meeting.
The PBI Network scoring form is based on the National Institutes of Health request for proposal scoring system and utilizes Likert-type criterion scores and weighted impact scores. A proposal’s final overall impact score is a total of these weighted scores and may range from 1 to 9 with 1 representing an exceptional score and 9 representing a poor score. Below is guidance for using the scoring form to calculate criterion scores, impact scores, and overall impact scores.
Criterion Scores:
Ratings utilize a 9-point Likert-type scale (described in Table I) and are in whole numbers only.
The entire range of the scale (1–9) should be considered.
The scale is used by the assigned reviewer to evaluate five individual criteria (i.e., Mission Alignment, Quality of Evidence, Solution Maturity, Established Need, and Feasibility of Implementation).
Reviewers should consider the strengths and weaknesses within each criterion. For example, a major strength may outweigh many minor and correctable weaknesses.
It is expected that scores of 1 or 9 be used less frequently than other scores.
A criterion score of 5 is a good, medium strength score.
Table I.Criterion Score Descriptors
Criterion Strength Score Descriptor High 1 Exceptional 2 Outstanding 3 Excellent Medium 4 Very Good 5 Good 6 Satisfactory Low 7 Fair 8 Marginal 9 Poor Criterion Strength Score Descriptor High 1 Exceptional 2 Outstanding 3 Excellent Medium 4 Very Good 5 Good 6 Satisfactory Low 7 Fair 8 Marginal 9 Poor Table I.Criterion Score Descriptors
Criterion Strength Score Descriptor High 1 Exceptional 2 Outstanding 3 Excellent Medium 4 Very Good 5 Good 6 Satisfactory Low 7 Fair 8 Marginal 9 Poor Criterion Strength Score Descriptor High 1 Exceptional 2 Outstanding 3 Excellent Medium 4 Very Good 5 Good 6 Satisfactory Low 7 Fair 8 Marginal 9 Poor PBI Network Proposal Prioritization Form
Proposal Title: Reviewer: Please indicate (Y/N) if reviewer is member of proposal team? Criteria Reviewer Comments Score (1–9) Impact Mission Alignment: The solution has the capacity to improve the lives of our nation’s service members, veterans and their families by advancing excellence in psychological health and traumatic brain injury prevention and care.
____ X .30 = ____ Quality of Evidence: Evidence presented from at least one well designed, adequately powered randomized controlled published trial or quasi-experimental study that has been peer reviewed. Quasi-experimental is defined as research that shares similarities with traditional experimental designs or randomized controlled trials but lacks the element of random assignment to treatment or control groups.
____ X .20 = ____ Solution Maturity: The product is in its near final form and has been tested in at least one population that may or may not represent the targeted demographic.
____ X .10 = ____ Established Need: The need for the proposed solution is established by pre-identified requirements (e.g., VA/DoD Clinical Practice Guidelines, Department of Defense Instructions, etc.) generated by various authorities (e.g., DoD, PHCoE, NAM and SAMHSA).
____ X .15 = ____ Feasibility of Implementation: The resources required for implementation are reasonable and likely available.
____ X .25 = ____ Overall Impact Score: Sum (Score X Impact): ____ Proposal Title: Reviewer: Please indicate (Y/N) if reviewer is member of proposal team? Criteria Reviewer Comments Score (1–9) Impact Mission Alignment: The solution has the capacity to improve the lives of our nation’s service members, veterans and their families by advancing excellence in psychological health and traumatic brain injury prevention and care.
____ X .30 = ____ Quality of Evidence: Evidence presented from at least one well designed, adequately powered randomized controlled published trial or quasi-experimental study that has been peer reviewed. Quasi-experimental is defined as research that shares similarities with traditional experimental designs or randomized controlled trials but lacks the element of random assignment to treatment or control groups.
____ X .20 = ____ Solution Maturity: The product is in its near final form and has been tested in at least one population that may or may not represent the targeted demographic.
____ X .10 = ____ Established Need: The need for the proposed solution is established by pre-identified requirements (e.g., VA/DoD Clinical Practice Guidelines, Department of Defense Instructions, etc.) generated by various authorities (e.g., DoD, PHCoE, NAM and SAMHSA).
____ X .15 = ____ Feasibility of Implementation: The resources required for implementation are reasonable and likely available.
____ X .25 = ____ Overall Impact Score: Sum (Score X Impact): ____ PBI Network Proposal Prioritization Form
Proposal Title: Reviewer: Please indicate (Y/N) if reviewer is member of proposal team? Criteria Reviewer Comments Score (1–9) Impact Mission Alignment: The solution has the capacity to improve the lives of our nation’s service members, veterans and their families by advancing excellence in psychological health and traumatic brain injury prevention and care.
____ X .30 = ____ Quality of Evidence: Evidence presented from at least one well designed, adequately powered randomized controlled published trial or quasi-experimental study that has been peer reviewed. Quasi-experimental is defined as research that shares similarities with traditional experimental designs or randomized controlled trials but lacks the element of random assignment to treatment or control groups.
____ X .20 = ____ Solution Maturity: The product is in its near final form and has been tested in at least one population that may or may not represent the targeted demographic.
____ X .10 = ____ Established Need: The need for the proposed solution is established by pre-identified requirements (e.g., VA/DoD Clinical Practice Guidelines, Department of Defense Instructions, etc.) generated by various authorities (e.g., DoD, PHCoE, NAM and SAMHSA).
____ X .15 = ____ Feasibility of Implementation: The resources required for implementation are reasonable and likely available.
____ X .25 = ____ Overall Impact Score: Sum (Score X Impact): ____ Proposal Title: Reviewer: Please indicate (Y/N) if reviewer is member of proposal team? Criteria Reviewer Comments Score (1–9) Impact Mission Alignment: The solution has the capacity to improve the lives of our nation’s service members, veterans and their families by advancing excellence in psychological health and traumatic brain injury prevention and care.
____ X .30 = ____ Quality of Evidence: Evidence presented from at least one well designed, adequately powered randomized controlled published trial or quasi-experimental study that has been peer reviewed. Quasi-experimental is defined as research that shares similarities with traditional experimental designs or randomized controlled trials but lacks the element of random assignment to treatment or control groups.
____ X .20 = ____ Solution Maturity: The product is in its near final form and has been tested in at least one population that may or may not represent the targeted demographic.
____ X .10 = ____ Established Need: The need for the proposed solution is established by pre-identified requirements (e.g., VA/DoD Clinical Practice Guidelines, Department of Defense Instructions, etc.) generated by various authorities (e.g., DoD, PHCoE, NAM and SAMHSA).
____ X .15 = ____ Feasibility of Implementation: The resources required for implementation are reasonable and likely available.
____ X .25 = ____ Overall Impact Score: Sum (Score X Impact): ____
Impact Scores:
Each criterion score is assigned a pre-determined weight, called an impact score, based upon PHCoE’s determination of the importance of each criteria.
The impact score for the proposal is not intended to be an average of criterion scores.
The impact score is based on the criterion score multiplied by the pre-determined weighted impact of that criteria.
Criteria scores will be multiplied by the impact factor. The total of these weighted scores will determine a proposal’s overall impact (i.e., final score).
Overall Impact Scores:
Overall impact is the proposed project’s likelihood to have a sustained, powerful influence on the Military Health System and the lives of service members and beneficiaries.
The overall impact score will reflect the overall evaluation, taking into account the assigned weight of the criterion score.
Since each criterion is assigned a specific weight, a proposal does not need to be strong in all categories to be judged likely to have major impact.
Reference material:
Defense Health Agency, Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury. (2016). Operational manual for conducting knowledge translation in the military health system. (Unpublished Draft). Silver Spring, MD: Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury.
U.S. Department of Health & Human Services, National Institutes of Health, Office of Extramural Research. (2013). Scoring System and Procedure. Available at https://grants.nih.gov/grants/peer/guidelines_general/scoring_system_and_procedure.pdf
References
Author notes
The views expressed in this paper are those of the authors and do not necessarily represent the official position or policy of the U.S. Government or the Department of Defense.
